Source-linked AI summary
Perceptio: Perception Enhanced Vision Language Models via Spatial Token Generation
Yuchen Li, Amanmeet Garg, Shalini Chaudhuri, Rui Zhao, Garin Kessler
TL;DR
LVLMs perform well on semantic multimodal tasks but struggle with fine-grained spatial grounding because they do not explicitly generate spatial interpretations. Perceptio inserts segmentation and depth tokens into the autoregressive sequence before text answers, achieving state-of-the-art results across referring segmentation, spatial reasoning, and VQA benchmarks. The authors identify trade-offs for text-only tasks, static-image scope, and dependence on frozen teacher models.
Problem
LVLMs struggle with fine-grained spatial grounding because spatial relationships are typically inferred without explicit spatial cues or perception outputs.
Method
Perceptio generates segmentation and discretized depth tokens before answer tokens, using joint perception-language training, composite depth objectives, and differentiable depth reconstruction.
Results
Perceptio achieves state-of-the-art performance across referring segmentation, depth understanding, and general VQA benchmarks, including a +10.3% HardBLINK improvement and +1.0% MMBench gain.
Takeaways & Limitations
Explicit in-sequence perception materially strengthens spatial grounding across diverse image understanding and question-answering tasks.
Takeaways & Limitations
The method is limited to static images, relies on frozen teacher models whose errors can propagate, and slightly improves some general VQA metrics when depth tokens are removed.
Abstract
from arXiv · showhide
Large Vision Language Models (LVLMs) excel at semantic understanding but struggle with fine grained spatial grounding, as the model must implicitly infer complex geometry without ever producing a spatial interpretation. We present Perceptio, a perception enhanced LVLM with 2D and 3D spatial reasoning abilities, enabled via explicit semantic segmentation tokens and depth tokens generated directly within the autoregressive sequence. Concretely, we (i) distill a VQVAE depth codebook from a strong monocular teacher to tokenize dense depth into compact sequences, and (ii) integrate SAM2 based semantic segmentation tokens and VQ-VAE depth tokens inside the LLM so the model first emits spatial tokens and then answers. To stabilize depth token generation, we introduce novel composite depth-token objectives (marker, token, and count losses) and a soft-merging technique for differentiable reconstruction. We adopt a multi-task co-training strategy across diverse datasets, letting the model learn perception tokens to tackle multiple downstream tasks. Building on InternVL, Perceptio achieves state-of-the-art performance across benchmarks: improving referring expression segmentation by +0.8/+1.4/+1.1 cIoU on RefCOCO/+/g HardBLINK spatial understanding accuracy by 10.3%, and MMBench accuracy by 1.0%, demonstrating that explicit spatial chain-of-thought materially strengthens spatial grounding in LVLMs.
1 Introduction
Perceptio addresses LVLMs’ weak spatial understanding by generating explicit 2D segmentation and 3D depth tokens before producing answers. It combines perception-enhanced training and a joint dataset to improve spatial grounding across segmentation, spatial reasoning, and VQA tasks.
- 1 Introduction: Perceptio jointly generates 2D semantic segmentation and discretized 3D depth tokens within an autoregressive LVLM sequence.The model emits perception tokens before text answers, using explicit spatial signals during generation.
- 1 Introduction: VQ-VAE depth tokens, segmentation tokens, and composite marker, token, and count losses provide explicit spatial perception and stabilize depth-token emission.The approach uses a depth codebook distilled from specialist depth predictions and a soft reconstruction technique for differentiable training.
- 1 Introduction: 56K examples pair segmentation masks and depth priors with language supervision, augmenting RefCOCO/+/g with aligned depth tokens and attribute descriptions.The dataset is designed for joint perception-language training across downstream tasks.
- 1 Introduction: Perceptio achieves state-of-the-art performance on RefCOCO/+/g, improves HardBLINK spatial reasoning by +10.3%, and improves MMBench accuracy by +1.0%.The reported gains span referring segmentation, spatial reasoning, and general VQA.
2 Related Work
Related work shows that LVLM spatial reasoning remains difficult because geometry is usually expected to emerge from large-scale training rather than being modeled explicitly. Existing approaches improve targeted perception but often separate dense spatial processing from the language reasoning loop.
- 2 Related Work: Modern LVLMs support multimodal understanding tasks including captioning, VQA, and grounding, while fine-grained spatial reasoning remains a persistent challenge.The related-work context contrasts broad multimodal capability with limitations in geometric understanding.
- 2.1 Large Vision-Language Models (LVLMs): LVLMs typically assume complex spatial relationships emerge from training at scale instead of treating spatial reasoning as a foundational objective.InternVL2.5-26B reaches only 33.1% average accuracy on HardBLINK’s closer-to-camera point-selection task.
- 2 Related Work: Two-stage and token-emitting systems improve segmentation or bring perception into the sequence, but prior approaches often externalize perception or provide incomplete spatial grounding.Specialist pipelines can excel on targeted spatial tasks while trading off broad conversational ability.
3 Methods
Perceptio integrates segmentation and discretized depth tokens into an autoregressive LVLM sequence before text generation, using joint supervision to produce spatially grounded answers. Its depth pathway combines VQ-VAE tokenization with specialized objectives for reliable token emission and reconstruction.
- Architecture: Perceptio routes image and query information through standard, segmentation-aware, and depth-codebook pathways before the LLM generates interleaved perception and language tokens.The system uses a standard image encoder, frozen SAM encoder, and frozen depth VQ-VAE codebook.
- Perception-Enhanced Generation: The enforced sequence emits a segmentation token block, a depth token block, and then the textual answer.The segmentation token conditions a query-grounded mask decoder, while discretized depth tokens represent the depth map.
- Depth Codebook: Depth maps from Depth Anything V2 are discretized through a VQ-VAE codebook into compact depth-token sequences integrated into the model vocabulary.The codebook is trained on depth maps from the same scene-image corpora used for LLM fine-tuning.
- Multi-Task Objective: Joint training combines standard language modeling, segmentation reconstruction, depth-token generation, and differentiable depth reconstruction losses.The total objective weights the depth-generation and depth-reconstruction terms with λd and λr.
- Depth Token Objectives: The composite depth-token loss aligns sequence boundaries, token values, and sequence length through marker, token, and count terms.Marker loss targets dstart and dend positions; token loss targets depth-code values; count loss targets the desired length n.
4 Results
Perceptio performs strongly across referring segmentation, image chat, and spatial reasoning benchmarks, while qualitative results show accurate joint segmentation and depth predictions. Ablations indicate that depth and segmentation signals contribute differently across spatial and general VQA tasks.
- Main Results: 82.7%, 77.9%, and 80.0% cIoU on RefCOCO, RefCOCO+, and RefCOCOg establish Perceptio-8B as state of the art across all three referring segmentation datasets.These results surpass Sa2VA-8B by +1.1/+1.7/+1.3 points, respectively.
- Main Results: 71.0 average accuracy on HardBLINK improves over LLaVA-Aurora by 10.3 points, with Perceptio-8B reaching 75.8/71.0/66.1 for 3/4/5 points.The corresponding gains are +8.9/+10.5/+11.3 points for the three point-count settings.
- Main Results: 83.4 MMBench accuracy and MME perception/cognition scores of 1654/628 show strong image-chat performance for Perceptio-8B.Perceptio-4B follows similar trends and outperforms some larger baselines on reported benchmarks.
- Qualitative Results: Qualitative examples show reconstructed depth maps with clear depth separation and segmentation masks aligned with object boundaries.Perceptio produces accurate depth and segmentation outputs on examples where Sa2VA fails to predict correct masks.
- Ablation Studies: Removing depth tokens reduces HardBLINK average accuracy from 71.0% to 45.2%, a decline of 25.8 percentage points, while general VQA metrics slightly improve.This ablation isolates the depth dataset, task, and loss function from the full model.
- Ablation Studies: Removing segmentation tokens lowers MME to 1620/585, MMBench by 1.6 points, and SEED-Bench by 2.3 points, indicating depth alone is insufficient for strong VQA reasoning.The results support complementary roles for geometric depth cues and explicit semantic grouping.
- Ablation Studies: Removing either depth-specific objective degrades benchmark results, with soft reconstruction supporting continuous depth fidelity and token loss sharpening discrete depth sequences.Without LDepthRecon, MMBench reaches 81.9%; without Ldepth, it reaches 82.4% and SEED-Bench falls from 75.7% to 74.3%.
5 Discussion and Conclusion
Perceptio integrates 2D segmentation and discretized 3D depth tokens into one autoregressive sequence, achieving strong spatial and general-task performance while exposing clear scope and optimization limitations.
- Perceptio emits segmentation and discretized depth tokens before text answers, forming an in-sequence spatial chain-of-thought.
- Perceptio achieves state-of-the-art results on referring expression segmentation, depth understanding, and general VQA benchmarks.
- Removing depth tokens collapses HardBLINK accuracy by 25.8 points, while removing segmentation tokens degrades SEED-Bench general VQA by up to 2.3 points.
- The model’s depth-token design introduces a mild optimization tension with text-only tasks, with removing depth tokens slightly improving MMBench by +0.4%.
- Training and evaluation are limited to static images, while frozen teacher models can propagate Depth Anything V2 and SAM2 errors to the student.
A Depth reconstruction loss
Perceptio makes depth reconstruction differentiable through soft token averaging, with the algorithmic details specified in Algorithm 1.
- Soft token averaging makes the depth reconstruction loss differentiable.
B 3D reasoning task examples
The HardBLINK reasoning task requires a 3D depth signal, illustrated through examples presented in Figure 6.
- HardBLINK examples illustrate a reasoning task that requires a 3D depth signal.
C 2D reasoning task examples
Perceptio examples combine depth reconstruction and segmentation outputs to visualize spatial reasoning and mask prediction on HardBLINK and RefCOCOg.
- Depth reconstruction procedure: Soft codebook mixing reconstructs depth by masking logits to depth-codebook indices, averaging codebook embeddings, and decoding the resulting representation.
- 3D reasoning task examples: Figure 6 presents HardBLINK examples with original images, reconstructed depth maps, and image-depth overlays.
- 3D reasoning task examples: The HardBLINK overlays show correct decisions aligned with the perceived depth map.
- 2D reasoning task examples: Figure 7 presents RefCOCOg samples with RGB images, ground-truth masks, predicted masks, and predicted-mask overlays.