PaperlayerView exact revision on arXiv

Source-linked AI summary

SpatialBoost: Enhancing Visual Representation through Language-Guided Reasoning

Byungwoo Jeon, Dongyoung Kim, Huiwon Jang, Insoo Kim, Jinwoo Shin

arXiv:2603.22057v1cs.CV

TL;DR

Pre-trained vision encoders often struggle to acquire 3D spatial awareness because they are predominantly trained on 2D images. SpatialBoost injects linguistically expressed spatial knowledge through LLM-guided multi-turn reasoning, consistently enhancing encoders across spatial and general vision tasks.

  • Problem

    Pre-trained vision encoders are predominantly trained on 2D images and face challenges acquiring 3D spatial awareness, while existing spatial VQA data provide limited information for comprehensive 3D understanding.

  • Method

    SpatialBoost injects linguistically expressed geometric and semantic information into vision encoders using LLMs, dual-channel attention, hierarchical spatial reasoning, and multi-turn CoT data for single- and multi-view images.

  • Results

    SpatialBoost consistently improves vision encoders on tasks requiring 3D spatial knowledge and across diverse downstream benchmarks; for example, it improves DINOv3 by 3.5%p on SQA3D, from 51.4% to 54.9%.

  • Takeaways & Limitations

    SpatialBoost enhances vision encoders’ spatial understanding across varied downstream tasks, supporting further research on designing and improving vision encoders.

  • Takeaways & Limitations

    The spatial reasoning dataset construction relies on vision models, and fully addressing this concern would require costly large-scale ground-truth spatial annotations.

Abstract

from arXiv · show

Despite the remarkable success of large-scale pre-trained image representation models (i.e., vision encoders) across various vision tasks, they are predominantly trained on 2D image data and therefore often fail to capture 3D spatial relationships between objects and backgrounds in the real world, constraining their effectiveness in many downstream applications. To address this, we propose SpatialBoost, a scalable framework that enhances the spatial awareness of existing pre-trained vision encoders by injecting 3D spatial knowledge expressed in linguistic descriptions. The core idea involves converting dense 3D spatial information from 2D images into linguistic expressions, which is then used to inject such spatial knowledge into vision encoders through a Large Language Model (LLM). To this end, we adopt a multi-turn Chain-of-Thought (CoT) reasoning process that progressively incorporates dense spatial knowledge and builds hierarchical spatial understanding. To validate effectiveness, we adapt SpatialBoost to state-of-the-art vision encoders such as DINOv3, and evaluate its performance gains on a wide range of benchmarks requiring both 3D perception and general vision abilities. For instance, SpatialBoost improves DINOv3 performance from 55.9 to 59.7 mIoU on ADE20K, achieving state-of-the-art performance with 3.8% gain over the pre-trained DINOv3.

1 Introduction

SpatialBoost addresses the limited 3D spatial awareness of predominantly 2D-trained vision encoders by injecting linguistically expressed spatial knowledge through language-guided reasoning. It combines hierarchical spatial reasoning with parameter-efficient encoder adaptation and improves performance across 3D and general vision benchmarks.

  • 2D-trained vision encoders struggle to discern 3D spatial relationships between objects and images’ backgrounds.
  • SpatialBoost converts spatial cues inferred from 2D images into explicit linguistic representations that capture dense geometric relationships.The approach leverages object positions, point depths, and language’s sequential, structured compositionality.
  • SpatialBoost injects language-described spatial knowledge into pre-trained encoders with LLM-based decoder fine-tuning and a dual-channel attention module.The additional learnable parameters are trained while existing encoder parameters remain frozen to preserve prior knowledge.
  • The multi-turn reasoning process builds hierarchical understanding through pixel-level, object-level, and scene-level questions and answers.
  • SpatialBoost consistently improves 3D spatial tasks and general vision benchmarks, including DINOv3 ImageNet linear probing from 88.4% to 90.2%.It also improves DINOv3 on SQA3D from 51.4% to 54.9% and SigLIPv2 NYUd linear probing RMSE from 0.51 to 0.39.

2 Related Work

Related work spans supervised and self-supervised image representation learning, multimodal vision-language modeling, and methods for learning 3D spatial representations from multi-view or video data.

  • Supervised representation learning relies on large labeled datasets, motivating self-supervised learning to reduce dependence on costly annotations.Contrastive methods learn from augmented views of the same image and different images.
  • Vision-language models jointly represent visual and textual information using contrastive, masked-prediction, or autoregressive objectives.
  • Recent 3D-focused vision tasks increasingly use multi-view images from different camera viewpoints or video sequences as spatially informative inputs.

3 Method

SpatialBoost combines a dual-channel vision-language architecture with hierarchical spatial Chain-of-Thought data built from single- and multi-view images. Its reasoning dataset progresses from pixel geometry to object relations and scene distances, while the training pipeline aligns visual and textual representations before encoder fine-tuning.

  • 3.1 Training Pipeline: SpatialBoost injects natural-language 3D spatial information into vision encoders through dual-channel attention that preserves original visual features.
  • 3.1 Training Pipeline: The training pipeline encodes images, projects visual features into the LLM token space, concatenates them with text tokens, and optimizes autoregressive supervised fine-tuning loss.
  • 3.1 Training Pipeline: Training proceeds through feature alignment, visual instruction tuning, and vision-encoder fine-tuning with dual-channel attention.The first stage trains only the projector, the second tunes the projector and LLM with VQA data, and the third trains the encoder and projector while freezing the LLM.
  • 3.1 Training Pipeline: The dual-channel layer merges original and additional attention outputs using a trainable mixture factor α.The method fine-tunes the additional attention block and α while keeping other parameters frozen.
  • 3.2 Enhancing Vision Encoder with Spatial CoT: SpatialBoost addresses simple single-turn spatial QA by constructing richer multi-turn Chain-of-Thought data for single-view and multi-view images.
  • 3.2 Enhancing Vision Encoder with Spatial CoT: The spatial reasoning dataset progresses from pixel-level 3D point positions to object-level relations and scene-level distances.Object-level reasoning uses pixel-level information as a rationale, while scene-level questions require coherent 3D spatial understanding.

4 Experiments

SpatialBoost is evaluated across spatial, general vision, and robot-learning benchmarks, with ablations examining data, reasoning order, attention, and scalability. The results report consistent gains in spatial understanding and improvements on several general vision tasks.

  • Experimental Setup: SpatialBoost evaluates depth estimation, segmentation, 3D scene understanding, robot learning, classification, and retrieval using multiple pretrained vision encoders.The baselines include OpenCLIP, SigLIPv2, DINOv2, DINOv3, V-JEPAv2, AIMv2, dino.txt, TIPS, and Perception Encoder.
  • Dense Prediction Tasks: DINOv3’s ADE20K mIoU increases from 55.9% to 59.7%, while OpenCLIP’s NYUd RMSE decreases from 0.53 to 0.40 with a linear head.These results cover both semantic and geometric dense prediction.
  • Complex 3D-centric Tasks: OpenCLIP’s ScanQA BLEU-1 rises from 36.9 to 39.2, and DINOv3’s SQA3D score increases from 51.4 to 54.9.OpenCLIP’s 3D semantic segmentation also rises from 6.9 to 54.9 mIoU.
  • Vision-based Robot Learning: DINOv2 + SpatialBoost achieves 68.1% on Adroit versus 55.4% for DINOv2 alone.The reported gains extend to vision-based robot control.
  • General Vision Tasks: ImageNet accuracy rises from 88.4% to 90.2%, while Oxford-Hard mAP increases from 60.7 to 64.1.These results indicate gains on classification and retrieval tasks that do not explicitly require spatial understanding.
  • Ablation Study and Analysis: Forward hierarchical reasoning achieves the optimal performance, and combined single-view plus multi-view training outperforms either data type alone.The ablations also report that naive post-training is ineffective, dual-channel attention preserves or enhances pretrained knowledge, and larger datasets yield consistent improvements.

5 Conclusion

The paper presents SpatialBoost as a framework for improving vision encoders by injecting geometric and semantic image information through language. Experiments report consistent improvements on downstream tasks requiring spatial understanding.

  • Conclusion: SpatialBoost injects linguistic expressions of geometric and semantic image information into vision encoders.The framework uses an LLM, dual-channel attention layers, and a multi-turn visual spatial reasoning dataset.
  • Conclusion: Experiments show consistent enhancement of vision encoders on downstream tasks requiring spatial understanding.The conclusion states that the framework is intended to facilitate future research on designing and enhancing vision encoders.

A.1 Training Details of Stage 1 & 2

The appendix describes training and evaluation procedures spanning multimodal pretraining, encoder fine-tuning, dense prediction, 3D understanding, robot learning, classification, and retrieval.

  • Training Details: The multimodal architecture uses Qwen-2.0-7B as the LLM backbone and a 2-layer MLP projector.Training uses BLIP-558K for feature alignment and LLaVA-Instruct-158K plus multi-view VQA data for visual instruction tuning.
  • Training Details: Dual-channel attention increases parameters by 30% for OpenCLIP and SigLIPv2 and by 25% for DINOv2 and DINOv3.The LLM decoder is frozen while the vision encoder and projector are fine-tuned on multi-turn spatial reasoning data.
  • Dense Prediction Tasks: Dense prediction evaluation uses NYUd and KITTI for monocular depth estimation and ADE20K and Pascal VOC for semantic segmentation.Linear or DPT heads are used for depth, and linear or multi-scale heads for segmentation; visual backbones remain frozen during training.
  • 3D Scene Understanding: Lexicon3D evaluation probes vision-language reasoning, visual grounding, 3D semantic segmentation, and geometric correspondence with frozen visual backbones and task-specific heads.The appendix details Q-Former, attention-based fusion, REGTR-style cross-encoding, and linear probing heads for these tasks.
  • Robot Learning: Robot agents use 100 demonstrations per task, keypoint augmentation, an end-effector controller with path planning, and 224×224 front-camera images without depth measurements.Performance is averaged over five evaluations reporting the mean of the best performance.
  • Classification and Retrieval: Classification uses a linear classifier on the final [CLS] token, while retrieval ranks images by cosine similarity between query and target [CLS] tokens.ImageNet classification follows DINOv3’s protocol, and retrieval evaluates Oxford, Paris, AmsterTime, and Met datasets.

B Multi-view VQA Dataset

The multi-view VQA dataset is constructed to supply richer 3D information and address the predominance of single-view VQA data. It combines selected multi-view pairs with several question types generated by GPT-4o.

  • Motivation: The dataset targets multi-view reasoning because existing VQA datasets largely focus on single-view scenarios.The authors state that proper instruction tuning is important for transferring 3D information to vision encoders.
  • Data Sources: Multi-view samples are drawn from ScanNet, Mip-NeRF360, MVImgNet, and Ego4D.The construction uses both 3D datasets and ego-centric video data.
  • Pair Selection: Image pairs are selected when their LPIPS distance satisfies 0.35 ≤ LPIPS(x_i, x_j) ≤ 0.65.This constraint filters outlier samples for meaningful multi-view learning.
  • Question Generation: GPT-4o generates common, adversarial, and multiple-choice VQA questions from selected image pairs.These question types probe general knowledge understanding from multi-view visual inputs.

C Multi-turn Visual Spatial Reasoning Dataset

The dataset converts single-view and multi-view visual information into 12-turn hierarchical spatial reasoning sequences, progressing from pixel-level depth to object- and scene-level 3D understanding. It uses specialized vision models and multi-view point clouds to generate spatial QA data, with viewpoint expansion for arbitrary multi-view configurations.

  • Hierarchical reasoning: The dataset associates each single-view or multi-view input with 12 sequential QA turns spanning pixel-, object-, scene-, and caption-level reasoning.The first five turns address point-wise depth, the next four approximate 3D bounding cubes, and later turns address holistic 3D interpretation and scene captions.
  • Hierarchical reasoning: Each turn builds on previous answers, enabling chain-of-thought reasoning that integrates pixel- and object-level details into scene-level spatial understanding.Depth and segmentation models provide the extracted 3D information used to synthesize the QA pairs.
  • Data generation: Single-view data uses depth and segmentation to construct 3D point clouds, while multi-view data uses VGGT reconstruction and canonical-space bounding cubes.Pixel-level questions sample coordinates from object bounding boxes, whereas object- and scene-level questions sample 3D bounding cubes.
  • QA design: The QA hierarchy includes point-depth queries, object bounding-cube and relative-position queries, and scene-level relative-distance queries.Depth values are rounded to three decimal places, with centimeter-scale values used below 0.5 meters.
  • Multi-view expansion: Interpolated viewpoints are retained when anchor-view QA pairs remain valid for more than half of them, expanding two-view inputs to arbitrary multi-view configurations.The 200K multi-view samples include 160K two-view, 30K four-view, and 10K eight-view samples.

D Details of Ablation Study

The ablation study tests whether LLM supervision and hierarchical reasoning improve SpatialBoost, comparing LLM-based fine-tuning with pixel-level alternatives and evaluating reasoning-level combinations across vision tasks. The results favor LLM supervision and the combination of all reasoning levels.

  • LLM-based supervision: LLM-based fine-tuning consistently outperforms linear-layer, SAM-decoder, and VGGT-decoder pixel-level alternatives across classification, segmentation, and depth estimation.The comparison evaluates ImageNet-1K classification, ADE20K segmentation, and NYUd depth estimation.
  • Experimental setup: The ablation fixes 300K reasoning-data samples for most comparisons and uses DINOv2 ViT-L/14 as the vision encoder architecture.The VGGT experiment is the stated exception to the fixed-sample setup.
  • Reasoning hierarchy: The reasoning-hierarchy study fixes 100K samples and evaluates combinations of pixel-, object-, and scene-level reasoning on depth, segmentation, and classification.All experiments use ViT-L/14 under otherwise consistent settings.
  • Reasoning hierarchy: The full combination of reasoning levels achieves the best performance across all evaluated tasks, supporting hierarchical reasoning.Pixel-level QA is especially strong for dense prediction, while object-level QA produces strong classification improvements.

E.2 Detailed Analysis on Single-view and Multi-view Data

Additional analyses examine multi-view data, dual-channel attention, scalability, and data-generation bias. Multi-view reasoning improves spatial tasks, dual-channel attention preserves broader performance, and the reported gains remain robust across scaling and supervision sources.

  • Single-view and multi-view data: Multi-view reasoning data improves depth estimation, segmentation, geometric understanding, and 3D semantic understanding.With increasing multi-view data, SigLIPv2 geometric-understanding registration recall rises from 84.1% to 86.4%, while 3D semantic-understanding mIoU rises from 51.8% to 55.5%.
  • Dual-channel attention: Dual-channel attention consistently improves classification, segmentation, and depth estimation, unlike full fine-tuning and LoRA, which show classification drops.The comparison uses DINOv2 ViT-L/14 with a fixed 100K reasoning-data sample size.
  • Data scalability: SpatialBoost improves SigLIPv2 and DINOv3 across all tasks in the detailed data-scalability results.The cited table is specifically described as covering classification, segmentation, and depth estimation.
  • Data-generation bias: Vision-foundation-model-based and ground-truth-based reasoning data produce negligible performance differences, indicating marginal bias propagation in the data pipeline.The comparison uses fixed 100K ScanNet single-view samples and evaluates classification, segmentation, depth, and ScanQA-related performance.

E.6 Additional Results on Multi-modal Large Language Models

SpatialBoost is applied to the vision encoders of InternVL3 and Qwen3-VL and evaluated through linear probing and diverse VQA tasks. It improves both encoder benchmarks and multimodal-language-model performance.

  • MLLM VQA performance: SpatialBoost also produces consistent gains across diverse VQA tasks for multimodal language models.Qwen3-VL scores rise from 72.8 to 76.4 on MUIRBench and from 48.8 to 51.5 on ERQA.

F Limitations

SpatialBoost’s spatial-reasoning data pipeline relies on vision models, creating a limitation that could ultimately require large-scale ground-truth spatial annotations to address.

  • The visual spatial reasoning dataset is constructed using vision models, so errors in those models remain a limitation of the pipeline.The authors report controlled experiments showing that these errors do not meaningfully propagate into the fine-tuned vision encoder.
  • Addressing this concern fundamentally would require large-scale ground-truth spatial annotations, whose collection and annotation remain costly and largely unexplored.
  • The authors identify curating large-scale spatial ground-truth data and developing more accurate vision foundation models as directions for extending the pipeline.
Loading 2603.22057v1…