PaperlayerView exact revision on arXiv

Source-linked AI summary

GeoGR: Spatial-Semantic Token Compression and Group Routing for Efficient 3D VLMs

Yongxin Zheng, Hao Wu, Fang Li, Hu Fu, Peiran Yin, Xudong Wang, Haozhe Hu, Xinghao Chen, Yunpu Ma, Wei Zhang, Xiaoyu Shen

arXiv:2610.13035v1cs.CVcs.AIcs.CL

TL;DR

Multi-view 3D VLMs face long, costly visual sequences with redundancy across views and LLM layers. GeoGR compresses scene representations and routes visual computation using recoverable token groups. It retains high fractions of dense-model performance while accelerating LLM prefill across models with and without explicit 3D spatial encoding.

  • Problem

    Multi-view 3D VLMs produce long visual-token sequences with repeated scene observations, while dense inference processes all visual tokens at every LLM layer.

    Source: §1 · p. 1

  • Method

    GeoGR combines voxel-based GeoSemZip compression with GroupRoute’s late entry, early exit, and recoverable group-wise skipping, supported by compression-aware post-training.

    Source: §1 · pp. 2–3

  • Results

    GeoGR retains 95.2% and 96.8% of dense post-trained model performance on LLaVA-OneVision-7B and Video-3D LLM-7B, with LLM prefill speedups of 6.8× and 6.1×, respectively.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Experiments show a favorable performance-efficiency trade-off across 3D VLMs with and without explicit 3D spatial encoding.

    Source: §6 · p. 11

Abstract

from arXiv · show

Multi-view representations enable pretrained image and video vision-language models to understand 3D scenes, but their long visual token sequences make inference costly. Redundancy arises both from repeated observations of the same scene regions across views and from processing all visual tokens at every LLM layer, regardless of their relevance to the current computation. We propose GeoGR, a two-stage framework that jointly compresses scene representations and visual computation. At the projector level, GeoSemZip consolidates cross-view observations into 3D voxel tokens and combines semantic saliency with spatial coverage to construct a compact scene representation. Unselected voxel features are aggregated into contextual anchors to retain scene information that would otherwise be discarded. At the LLM level, we identify an effective visual computation window and observe that query-conditioned token relevance remains locally stable but changes over longer layer intervals. Guided by these findings, GroupRoute combines late entry and early exit with recoverable group-wise skipping. Low-relevance tokens bypass complete LLM layers while retaining their hidden states and can rejoin the active group at subsequent anchor layers selected under a computation budget. For compression-aware post-training, we construct a compact dataset of 67K samples by retaining 30% of the original training pool. With this protocol, GeoGR retains 95.2% and 96.8% of the dense post-trained models' performance on LLaVA-OneVision-7B and Video-3D LLM-7B, respectively, while accelerating LLM prefill by $6.8\times$ and $6.1\times$. These results demonstrate the consistent effectiveness of GeoGR across 3D VLMs with and without explicit 3D spatial encoding, highlighting a favorable performance-efficiency trade-off. The code is available at https://github.com/EIT-NLP/GeoGR.

1 Introduction

Multi-view 3D VLMs face redundant scene tokens and costly layer-by-layer visual computation. GeoGR combines projector-level scene compression with query-conditioned, recoverable computation routing, retaining strong performance with faster prefill.

1 Introduction

  • GeoGR addresses redundancy from repeated observations across views and from processing every visual token at every LLM layer, despite shifting query relevance.Long visual sequences also make inference substantially more expensive; Video-3D LLM uses 32 views and 6,272 visual tokens per scene.Source: §1 · pp. 1–2
  • GeoSemZip consolidates cross-view observations into voxel tokens, selecting salient tokens and spatially distributed contextual anchors while aggregating unselected features by similarity.The anchors preserve information from less salient regions under compression.Source: §1 · p. 2
  • GroupRoute combines Late Entry, Early Exit, and recoverable group-wise skipping, preserving skipped tokens’ hidden states so they can rejoin at later anchor layers.BARS selects anchor positions under a computation budget to limit the risk of reusing groups.Source: §1 · p. 2
  • GeoGR retains 95.2% and 96.8% of the dense post-trained models’ performance on LLaVA-OneVision-7B and Video-3D LLM-7B, with LLM prefill speedups of 6.8× and 6.1×, respectively.The models represent 3D scenes without and with explicit 3D spatial encoding, respectively.Source: §1 · p. 3
  • Compression-aware post-training uses PRISM [2] to select the least redundant 30% of the original training pool, yielding 67K samples.The evaluation covers two question-answering benchmarks for LLaVA-OneVision and five benchmarks spanning question answering, visual grounding, and dense captioning for Video-3D LLM.Source: §1 · p. 3
  • GeoGR combines geometric consolidation, semantic saliency, and spatial coverage with computation routing that retains temporarily skipped tokens for later reactivation.The paper describes the framework as achieving a favorable performance-efficiency trade-off across the two evaluated 3D VLMs.Source: §1 · p. 3

2 Related Work

Prior work reduces visual tokens through saliency, merging, or geometry-aware selection, and reduces LLM computation through within-layer or depth-wise methods. GeoGR’s modules combine complementary strategies for multi-view 3D inputs.

2 Related Work

  • GeoSemZip combines salient-token selection with spatial coverage and aggregates unselected voxel features into contextual anchors.Prior multi-view methods use geometric correspondence, spatial diversity, voxel constraints, depth-derived tokens, or adaptive sampling; HiSC combines spatial graph merging with hierarchical pruning.Source: §2 · p. 3
  • GroupRoute extends late entry and early exit to 3D VLMs with query-conditioned groups that skip complete LLM layers and can be reassessed for reactivation.Related approaches reduce computation within attention or FFNs, skip or freeze visual updates, or prune tokens progressively.Source: §2 · p. 3

3 Understanding Layer-Wise Visual Token Dynamics in 3D VLMs

The section examines whether query-conditioned visual-token rankings remain stable across layers and whether initially low-ranked tokens later become important. Its findings motivate short-interval group reuse with reassessment rather than permanent removal.

3 Understanding Layer-Wise Visual Token Dynamics in 3D VLMs

  • The analysis tracks attention-based rankings across layers 8–23 on ScanQA [1] using LLaVA-OneVision, comparing layer pairs at distances of 1–10.It averages attention from all heads for the last prompt token before answer generation and measures rank consistency, rank shift, transitions, and attention share.Source: §3 · p. 4
  • Ranking stability decreases with layer distance, while initially low-ranked tokens increasingly enter the top half and account for more of its attention.Figure 2 summarizes both patterns.Source: Figure 2 · p. 4, §3 · p. 4
  • Nearby layers show higher rank correlation and smaller normalized rank shifts, supporting reuse of a token partition over short intervals.Agreement generally weakens as layer distance increases, so rankings cannot be assumed stable throughout the visual computation window.Source: §3 · p. 4
  • Initially low-ranked tokens can later enter the top half and contribute a growing, non-negligible share of its attention, so low rank does not guarantee lasting irrelevance.The section uses this finding to motivate temporary inactivity and later reassessment rather than permanent pruning.Source: §3 · pp. 4–5

4 Method

GeoGR combines projector-level scene-token compression with selective visual computation in the LLM. Its method also uses a compact post-training dataset to adapt models to compression.

4 Method

  • The method combines the two compression stages: GeoSemZip prepares compact visual tokens, and GroupRoute allocates their computation alongside always-active text tokens.The compressed visual tokens are combined with text tokens as LLM input; text tokens remain active throughout the LLM.Source: §4 · p. 5, Figure 3 · p. 6

4.1 Post-Training Protocol for 3D VLM Compression

  • Retaining 30% of samples yields a 67K-example dataset; dense post-trained Video-3D LLM-7B and LLaVA-OneVision-7B retain 98.72% and 95.69% of full-data performance, respectively.The dataset is distilled with PRISM, and the reported retention compares dense models post-trained on the distilled and full datasets.Source: §4.1 · p. 5, Table 1 · p. 5

4.2 GeoSemZip: Spatial–Semantic Token Compression

  • GeoSemZip consolidates cross-view observations into voxel tokens, allocates the token budget to salient dominant tokens and spatially distributed anchors, and merges unselected features into similar anchors.Anchors are sampled in Morton order, and merged residuals retain unselected content without increasing the token budget.Source: §4.2 · pp. 5–7, Figure 3 · p. 6

4.3 GroupRoute: Late Entry, Early Exit, and Recoverable Group-wise Skipping

  • GroupRoute limits visual processing to an intermediate window, routing query-relevant tokens through layers while skipped tokens retain their states and can rejoin at subsequent anchors.BARS selects anchor positions under a computation budget; the resulting schedule is fixed at inference while token groups remain query-dependent.Source: §4.3 · pp. 7–8

5 Experiments

Experiments compare GeoGR with compression baselines on two multi-view 3D VLMs and ablate its components. The results show strong performance retention alongside faster inference, with benefits across the tested models and components.

5.1 Experimental setup

  • The evaluation covers LLaVA-OneVision without explicit 3D representations and Video-3D LLM with depth- and pose-derived 3D embeddings, using 32 RGB views per scene.LLaVA-OneVision is evaluated on two QA benchmarks; Video-3D LLM is evaluated on five benchmarks spanning QA, visual grounding, and dense captioning.Source: §5.1 · p. 8

5.2 Main results

  • GeoGR retains 95.18% of dense-model performance on LLaVA-OneVision and 96.80% on Video-3D LLM, achieving the highest average retention at comparable computational budgets.On LLaVA-OneVision, it exceeds DTC by 1.88 percentage points at 8.65 versus 9.36 TFLOPs; on Video-3D LLM, it exceeds VisionZip† by 0.79 percentage points while reducing FLOPs by 3.2%.Source: §5.2 · p. 9, Table 2 · p. 9
  • LLM prefill speeds up 6.8× on LLaVA-OneVision and 6.1× on Video-3D LLM, while GeoGR offers a favorable performance-retention and inference-efficiency trade-off.The visual-token KV cache falls from 359.7 MB to 36.6 and 38.5 MB, respectively; projector latency is 99.0 and 75.6 ms.Source: §5.2 · p. 10, Table 3 · p. 10

5.3 Ablation studies

  • At a fixed 30% token budget, combining spatial coverage, semantic saliency, and voxel consolidation reaches 98.19% retention; residual merging raises it to 99.33%.Spatial coverage alone reaches 93.53%, semantic saliency alone reaches 96.99%, and residual merging adds 1.14 points without increasing the token budget.Source: §5.3 · p. 10, Table 4 · p. 10
  • Query-conditioned grouping raises average retention from 92.73% to 95.18% compared with reusing the projector-derived token ordering.The variants share the same execution schedule; query-conditioned grouping ranks tokens at each anchor using attention from the final prompt token.Source: §5.3 · p. 11, Table 5 · p. 11
  • Recoverable skipping at layers 9 and 13 achieves 95.18% retention, exceeding all tested pruning configurations at a matched average token budget.Skipped tokens preserve their hidden states and can rejoin at the later anchor, whereas tokens pruned at layer 9 remain unavailable at layer 13.Source: §5.3 · p. 11, Table 6 · p. 11

6 Conclusion

GeoGR combines projector-level scene compression with selective visual computation in the LLM. Experiments show a favorable performance–efficiency trade-off across 3D VLMs with and without explicit 3D spatial encoding.

6 Conclusion

  • GeoSemZip consolidates cross-view observations while preserving salient content, spatial coverage, and information from unselected regions.It aggregates unselected voxel features into contextual anchors.Source: §6 · p. 11
  • GroupRoute combines Late Entry, Early Exit, and Recoverable Group-wise Skipping, allowing temporarily inactive tokens to resume computation.The approach is guided by effective visual computation windows and locally stable but evolving token relevance.Source: §6 · p. 11
  • Experiments demonstrate a favorable performance–efficiency trade-off across 3D VLMs with and without explicit 3D spatial encoding.A compact selected dataset supports compression-aware post-training.Source: §6 · p. 11

A.1 Training and Evaluation Protocol

The protocol specifies view-feature construction, compression-aware post-training, and benchmark evaluation; the decoder schedule confines visual-token computation to blocks 8–23 with regrouping at blocks 9 and 13.

A.1 Training and Evaluation Protocol

  • For each of 32 sampled RGB views, projector features and encoder attention are pooled to a 14 × 14 grid, while depth and camera data reconstruct world coordinates.Evaluation normalizes ScanQA predictions and references with the benchmark cleaner; SQA3D uses benchmark-specific normalization and exact match.Source: §A.1 · p. 15
  • Post-training uses projector adaptation followed by late-entry/early-exit adaptation, with one epoch per stage and the vision encoder frozen throughout.The projector is updated only in stage one, while the LLM backbone is updated in both stages.Source: §A.1 · p. 15

A.2 Exact GroupRoute Execution Schedule in the LLM Decoder

  • Visual tokens participate in blocks 8–23, with full-token regrouping anchors at blocks 9 and 13 using keep ratio κ = 0.5.Each anchor recomputes the partition from its block-input representations for subsequent blocks.Source: §A.2 · p. 15, Table 7 · p. 15
  • Blocks 8–9 process all retained visual tokens; under anchor-9, only L9 bypasses computation in blocks 10–12, while block 13 processes all tokens and forms a new partition; under anchor-13, only L13 bypasses in blocks 14–23.Blocks 1–7 and 24–28 contain no visual tokens because of late entry and early exit.Source: §A.2 · p. 16

B More Implementation Details

BARS chooses regrouping anchors under a computation budget by evaluating query-conditioned token-group risks across layers. The evaluated 3D VLMs differ in whether spatial information is explicitly encoded in their visual representations.

B.1 Budget-Aware Anchor Selection (BARS)

  • BARS constructs query-conditioned high- and low-priority token groups from normalized, head-averaged attention scores at anchor layers.The query is the last prompt token for question answering and captioning, or the grounding token for localization.Source: §B.1 · p. 16
  • Reusing an anchor partition is scored by bypassed attention mass, missed high-priority tokens, and normalized rank displacement.Earlier skipped blocks receive greater weight in the segment risk.Source: §B.1 · p. 16
  • Dynamic programming retains the lowest-risk feasible predecessor for each boundary, cost, and anchor count, then backtracks candidate schedules.The search selects a larger anchor count only when risk falls and every dataset meets the relative-risk-reduction threshold τ = 5%.Source: §B.1 · p. 17
  • The visual-token budget counts the full-token prefix and anchors separately from the grouped blocks that reuse partitions.BARS selects anchor positions under fixed compression settings and a computation budget.Source: §B.1 · pp. 16–17

C.1 3D VLMs

  • LLaVA-OneVision uses RGB visual features, whereas Video-3D LLM encodes spatial coordinates derived from depth and camera parameters into its visual representations.Video-3D LLM also supports spatial reasoning tasks including object-conditioned captioning and visual grounding.Source: §C.1 · p. 18

C.2 Benchmarks and Datasets

Evaluation draws on five ScanNet-based benchmarks spanning question answering, dense captioning, and language grounding. The datasets differ in task demands and evaluation metrics.

C.2 Benchmarks and Datasets

  • The five ScanNet-based datasets provide post-training examples and evaluation benchmarks for 3D question answering, object captioning, and language grounding.LLaVA-OneVision is evaluated on the two question-answering benchmarks, while Video-3D LLM is evaluated on all five.Source: §C.2 · p. 18
  • ScanQA tests indoor-scene object, attribute, and spatial-relation questions using CIDEr and normalized exact match.Evaluation uses the validation split.Source: §C.2 · p. 19
  • SQA3D adds an agent’s described position and orientation, requiring answers from that situated perspective and evaluation of scene layout and spatial relations.The test split is evaluated with normalized EM.Source: §C.2 · p. 19
  • Scan2Cap evaluates captions for localized objects, with BLEU-4 and CIDEr reported at a 3D IoU threshold of 0.5.Captions may describe object attributes and surrounding objects.Source: §C.2 · p. 19
  • ScanRefer measures single-object localization from referring expressions using Acc@0.25 and Acc@0.5.Descriptions use appearance and relational cues to distinguish targets from nearby objects.Source: §C.2 · pp. 19–20
  • Multi3DRefer covers expressions referring to zero, one, or multiple objects and uses set-level F1 at 3D IoU thresholds of 0.25 and 0.5.Its evaluation includes a zero-target convention.Source: §C.2 · pp. 19–20

D.1 Effect of Data Selection on Post-Training Performance

The PRISM-selected 30% training subset preserves strong dense-model performance, while GeoSemZip’s budget and parameter studies examine token allocation, baseline comparisons, and sensitivity to design choices.

D.1 Effect of Data Selection on Post-Training Performance

  • 98.72% performance retention for Video-3D LLM and 95.69% for LLaVA-OneVision result from post-training on the PRISM-selected 30% subset.Retention is the unweighted mean of metric-wise ratios to the corresponding full-data post-trained checkpoint.Source: §D.1 · p. 20, Table 10 · p. 20

D.2 GeoSemZip Budget Selection and Baseline Comparison

  • 99.33% retention at 30% budget exceeds the 97.26% at 20%; increasing to 40% uses 593.74 more tokens and gives 98.38% retention.The 30% checkpoint is used as GroupRoute’s input.Source: §D.2 · pp. 20–21, Table 11 · p. 20
  • At approximately 10%, 20%, and 30% token budgets, GeoSemZip exceeds the strongest baseline by 1.39, 2.55, and 1.08 percentage points, respectively.The comparison is among post-trained projector-level compression methods.Source: §D.2 · p. 21, Table 12 · p. 21

D.3 GeoSemZip Parameter Sensitivity

  • At a 30% token budget, voxel width s = 0.1 gives the best tested trade-off, with 3,574 occupied voxels per scene on average.Smaller voxels leave duplication unresolved, while larger ones begin fusing surfaces from different objects.Source: §D.3 · p. 21, Table 13 · p. 21
  • Mean pooling lowers Avg. by 0.27 points relative to max pooling at the same token budget, so GeoSemZip uses max pooling.Max pooling lets the best-scoring view determine a voxel’s rank.Source: §D.3 · p. 22, Table 14 · p. 22
  • Deviating from the optimal budget split lowers the average metric by 0.34 points at ρ = 0.75, 1.10 at ρ = 0.65, and 0.81 at ρ = 0.95.The study finds that most of the budget should go to dominant tokens, with a smaller portion reserved for scene coverage.Source: §D.3 · p. 22, Table 15 · p. 22

D.4 Visual-Computation Window Selection

The selected LLaVA-OneVision computation window is [8] [23], and equal keep ratios at anchors 9 and 13 yield the highest tested average retention.

D.4 Visual-Computation Window Selection

  • 95.43% average performance: exiting after block 23 is best, exceeding exits after blocks 22 and 24 by 4.46 and 3.37 percentage points, respectively.The resulting window is [8] [23].Source: Table 16 · p. 22, §D.4 · pp. 22–23
  • Entry at block 8 achieves 93.53% average performance, only 0.46 percentage points below block 5 while reducing visual computation for three additional blocks.The late-entry and early-exit boundaries were selected sequentially, with the entry position fixed before comparing exit layers.Source: §D.4 · p. 22, Table 16 · p. 22

D.5 Keep-Ratio Allocation Across Anchors

  • Equal keep ratios (0.50, 0.50) yield the highest average retention (95.18%) among tested allocations at anchors 9 and 13.The two ratios govern different numbers of blocks and therefore lead to different computational costs.Source: §D.5 · p. 23

E Additional Analysis

Nearby layers preserve visual-token relevance rankings and high-relevance membership, while two-stage compression retains more dense-model performance than projector-only compression at a similar active-token count.

E.1 Layer-Pair visual token Relevance Stability

  • Nearby layer pairs show the strongest agreement in visual-token rankings and top-200 membership, while top-200 composition changes more across distant layers.This pattern motivates temporarily skipping low-priority tokens and regrouping all visual tokens at later anchors.Source: Figure 5 · p. 23, §E.1 · pp. 23–24

E.2 Complementarity of Projector-Level and LLM-Level Compression

  • 95.18% dense-performance retention at 638 active visual tokens per block: GeoGR exceeds projector-only compression’s 94.00% at 628 tokens by 1.18 points.GroupRoute-only retains 94.65% while processing 2,128 active tokens per block.Source: Table 18 · p. 24, §E.2 · p. 24
Loading 2610.13035v1…