PaperlayerView exact revision on arXiv

Source-linked AI summary

Zhengrong-Spatial-Aware 1.0: Advancing 3D Multimodal Intelligence

Yu Zhang, Xiaogang Wang, Yuhang Cheng, Jingrui Li, Wenbo Chen

arXiv:2610.37216v1cs.CV

TL;DR

Reliable metric 3D reasoning remains challenging when vision-language models observe only 2D images, despite their strengths in semantic recognition, captioning, and visual question answering. The work builds and validates complementary adaptation pipelines, combining HiSpatial metric-point-map processing with mixed-domain training and Qwen3-VL QLoRA. HiSpatial target accuracy improves from 1.73% to 79.89%; under mixed-domain training, Qwen3-VL-8B improves from 20.74% to 45.27% and Qwen3-VL-32B from 30.79% to 47.60%.

  • Problem

    Reliable metric 3D reasoning remains challenging when vision-language models observe only 2D images, despite their strengths in semantic recognition, captioning, and visual question answering.

    Source: §I · p. 1

  • Method

    The work builds and validates complementary adaptation pipelines, combining HiSpatial metric-point-map processing with mixed-domain training and Qwen3-VL QLoRA.

    Source: §I · p. 1, A. HiSpatial: Hierarchical 3D Spatial… · p. 2, B. HiSpatial Data Preparation · p. 3

  • Results

    HiSpatial target accuracy improves from 1.73% to 79.89%; under mixed-domain training, Qwen3-VL-8B improves from 20.74% to 45.27% and Qwen3-VL-32B from 30.79% to 47.60%.

    Source: §I · p. 1, §VII · p. 6

  • Takeaways & Limitations

    The pipelines provide a reproducible engineering foundation for further study of metric 3D reasoning, spatial grounding, multimodal data mixing, and geometry-aware model adaptation.

    Source: §VIII · p. 6

Abstract

from arXiv · show

This report presents the implementation and validation of two end-to-end multimodal adaptation pipelines for 3D-aware visual question answering on a custom spatial dataset. The first pipeline fine-tunes HiSpatial-3B, which augments RGB observations with metric-scale 3D point maps; the second adapts Qwen3-VL through memory-efficient QLoRA-based supervised fine-tuning. The completed engineering stack covers dataset conversion, image-annotation alignment, monocular geometry preprocessing, instruction construction, mixed-domain training, checkpoint management, batch inference, and automated evaluation. On the target spatial benchmark, HiSpatial adaptation increases overall accuracy from 1.73% before fine-tuning to 79.89% at 3,600 steps. On a separately collected styled split, accuracy improves from 14.9% to 55.1%, indicating transfer beyond the original training set. The Qwen3-VL pipeline is likewise implemented end-to-end for both 8B- and 32B-scale models using mixed-domain QLoRA adaptation. The resulting accuracy improves from 20.74% to 45.27% for Qwen3-VL-8B and from 30.79% to 47.60% for Qwen3-VL-32B. Together, these results establish two complementary and reproducible adaptation pipelines for geometry-aware and general-purpose multimodal spatial reasoning. These results establish a reproducible engineering foundation for subsequent work on metric 3D reasoning, spatial grounding, and domain-adapted multimodal systems, while also identifying training-data composition as a central factor in preserving general multimodal capability during specialization.

I. Introduction

The work addresses the challenge of reliable metric 3D reasoning from 2D images by building and validating practical adaptation pipelines for existing multimodal models. Its engineering outcomes cover data preparation, model adaptation, evaluation, and diagnosis of narrow-domain overspecialization.

I. Introduction

  • Reliable metric 3D reasoning remains challenging when vision-language models observe only 2D images, despite strengths in semantic recognition, captioning, and visual question answering.The target tasks include counting, height estimation, footprint comparison, distance estimation, spatial ordering, and relative direction.Source: §I · p. 1
  • The project builds a practical adaptation stack for existing open-source multimodal models rather than training a new foundation model from scratch.The reported engineering outcomes include aligned image–question–answer records, HiSpatial RGB+3D preprocessing, Qwen3-VL QLoRA pipelines, and a unified prediction/evaluation interface.Source: §I · p. 1
  • The reported combination of mixed-domain HiSpatial supervised fine-tuning, checkpointed inference, and automated evaluation improved target accuracy from 1.73% to 79.89%, according to the cited passage.Source: §I · p. 1
  • An empirical diagnosis of narrow-domain Qwen3-VL overspecialization motivates mixed-domain sampling and stronger retention controls in later experiments.The diagnosis is presented as motivation for subsequent experiments.Source: §I · p. 1

II. Model Principles and Literature Background

The background frames 3D spatial intelligence as a hierarchy of geometric, object-level, relational, and abstract reasoning. HiSpatial supplements visual appearance with explicit geometry to support questions grounded in physical quantities and relations.

A. HiSpatial: Hierarchical 3D Spatial Understanding

  • HiSpatial organizes 3D spatial intelligence into four levels, from geometric perception and object understanding through inter-object relations to abstract spatial reasoning.Examples range from depth or point queries to perspective taking and multi-step spatial problem solving.Source: A. HiSpatial: Hierarchical 3D Spatial… · p. 1
  • HiSpatial supplements RGB appearance with a point map whose pixels carry 3D coordinates, using metric-scale geometry for tasks requiring quantitative physical-unit reasoning.The point map is produced by monocular geometry estimation during training and inference; the cited MoGe-style formulation predicts an affine-invariant point map from one image [3].Source: A. HiSpatial: Hierarchical 3D Spatial… · pp. 1–2
  • HiSpatial-3B builds on PaliGemma2-3B and conditions visual representations on both appearance and geometry for supervision defined by geometric quantities and relations.PaliGemma 2 combines a SigLIP vision encoder and Gemma 2 language model [4].Source: A. HiSpatial: Hierarchical 3D Spatial… · p. 2

B. Qwen3-VL: General Multimodal Reasoning with 3D Grounding

Qwen3-VL serves as the general multimodal counterpart to HiSpatial, combining visual-language processing with spatial and 3D grounding. The project adapts it with QLoRA, while the two model branches provide distinct ways to approach spatial reasoning.

B. Qwen3-VL: General Multimodal Reasoning with 3D Grounding

  • Qwen3-VL combines a vision encoder, an MLP-based vision–language merger, and a Qwen3 decoder, with MRoPE and DeepStack supporting spatial-temporal position modeling and multi-level visual-feature injection [2].Its training recipe includes spatial reasoning and 3D grounding data.Source: B. Qwen3-VL: General Multimodal… · p. 2
  • The project adapts Qwen3-VL with QLoRA, keeping pretrained weights in low precision while training low-rank adapters to reduce memory use for 8B and 32B models.The prose identifies A and B as the trainable low-rank matrices.Source: B. Qwen3-VL: General Multimodal… · p. 2
  • HiSpatial supplies explicit input geometry, whereas Qwen3-VL relies primarily on learned visual-language representations and spatial priors, enabling evaluation of distinct sources of pipeline limitations.The stated factors include geometric reconstruction, domain adaptation, instruction formatting, and language-side reasoning.Source: B. Qwen3-VL: General Multimodal… · p. 2

A. Dataset Organization

The dataset pipeline organizes varied visual inputs and VQA supervision around deterministic sample identifiers. The benchmark groups spatial questions by capability and uses task-aware scoring, while the end-to-end workflow sends shared records through two model-specific adaptation branches and common evaluation.

A. Dataset Organization

  • The dataset supports raw and styled RGB images, overlays, masks, camera-pose metadata, scene annotations, and JSONL VQA supervision, with CAD/GLB assets available for building scenes.Sample identifiers map consistently to visual inputs for training, inference, and checkpoint evaluation.Source: A. Dataset Organization · p. 2

B. Spatial Question Types

  • The benchmark groups twelve question categories by evaluated capability and scores counting, metric estimates, and open-form relational answers with task-specific criteria.Metric answers use numerical parsing with relative tolerance, while open-form answers are normalized and evaluated semantically.Source: B. Spatial Question Types · p. 2

A. End-to-End System Overview

  • A shared data layer validates image–annotation alignment before the HiSpatial branch adds dense point maps and the Qwen3-VL branch converts supervision into conversational instructions; both feed common checkpointed inference and evaluation.The reference architectures pair HiSpatial RGB appearance with metric-scale point maps and Qwen3-VL visual features with a decoder and DeepStack injection [1], [2].Source: A. End-to-End System Overview · pp. 2–3, Figure 1 · p. 3

B. HiSpatial Data Preparation

HiSpatial data preparation synchronizes RGB images, point maps, and VQA records, then mixes target-domain data with general visual instructions. Checkpoint evaluation automates standardized prediction export and supports alternate visual inputs.

B. HiSpatial Data Preparation

  • HiSpatial preparation aligns scene images with VQA records, estimates dense XYZ tensors with MoGe, serializes synchronized samples, and mixes target data with general visual-instruction samples.The first successful setting used target building data and LLaVA-NeXT-style samples at approximately a 1:1 ratio.Source: B. HiSpatial Data Preparation · p. 3
  • Concentrated spatial supervision can teach the target answer format while narrowing a general VLM’s response distribution.The reported experiments motivate mixing general visual instruction data with target-domain samples.Source: B. HiSpatial Data Preparation · p. 3
  • Precomputed point maps are serialized with the VQA conversation, decoupling geometry estimation from model optimization and reducing repeated preprocessing cost.The passage also associates this design with improved experiment reproducibility.Source: B. HiSpatial Data Preparation · p. 3

C. HiSpatial Fine-Tuning and Checkpoint Evaluation

  • Fine-tuning starts from the released HiSpatial-3B checkpoint; the initial validated configuration used 1,200 steps and checkpointed every 400 steps.The configuration used per-device batch size 4 and effective total batch size 32; later runs extended training and adjusted the mixture.Source: C. HiSpatial Fine-Tuning and… · p. 3
  • Automated evaluation loads selected checkpoints, evaluates JSONL test splits, exports standardized predictions, and supports styled-image and colored-box variants without changing the protocol.The training launcher is provided through a project-level wrapper.Source: C. HiSpatial Fine-Tuning and… · p. 4

D. Qwen3-VL Instruction Construction and QLoRA

Qwen3-VL instruction records pair image inputs with user questions and assistant answers. The adaptation uses quantized low-rank updates to reduce memory needs for 8B- and 32B-scale models.

D. Qwen3-VL Instruction Construction and QLoRA

  • Qwen3-VL records use a multimodal conversational format, pairing an image with a user question and an assistant answer.The example shows the image referenced in the record and the question containing an image token.Source: D. Qwen3-VL Instruction Construction… · p. 4
  • The first experiment split 1,000 scenes into 900 training and 100 testing scenes, yielding approximately 10,800 training and 1,200 test QA instances.Each scene contributes 12 question types.Source: D. Qwen3-VL Instruction Construction… · p. 4
  • The Qwen adaptation uses 4-bit quantization and low-rank adapters; the vision encoder stays frozen while LoRA parameters are optimized in language-side projections.The multimodal connector can also be enabled depending on configuration, avoiding gradients for the full pretrained parameter set.Source: D. Qwen3-VL Instruction Construction… · p. 4

E. Unified Inference and Evaluation

The work provides a shared inference and evaluation interface for multiple spatial VLMs and reports HiSpatial results across reproduction checks and target adaptation. Target accuracy rises with training, while answer-format mixture affects performance on the original free-form task.

E. Unified Inference and Evaluation

  • A unified runner lets HiSpatial, Qwen3-VL, LLaVA-3D, and SpatialRGPT use the same JSONL benchmark, reducing model-specific evaluation scripts and making comparisons more reproducible.The interface maps sample identifiers to images and applies task-specific scoring for counting, metric estimates, and open-form spatial answers.Source: E. Unified Inference and Evaluation · p. 4

A. HiSpatial Reproduction Sanity Check

  • The reproduced HiSpatial stack scored 94.17% on CV-Bench-3D, 94.77% on CV-Bench Relation, 73.90% on EmbSpatial, and 78.05% on RoboSpatial as an implementation sanity check.These measurements used substantially less compute than the original training setup and were not presented as full reproduction of that regime.Source: A. HiSpatial Reproduction Sanity Check · p. 4

B. Target-Domain HiSpatial Adaptation

  • 1.73% before adaptation rose to 64.69% after 1,200 steps, 73.90% after 2,400 steps, and 79.89% after 3,600 steps on the original building-centric benchmark.On the separate styled dataset, accuracy increased from 14.9% for the released model to 49.2%, 52.8%, and 55.1% at those checkpoints.Source: B. Target-Domain HiSpatial Adaptation · p. 4, Table II · p. 5
  • A later data-mixture experiment found that naively combining free-form and multiple-choice supervision caused a pronounced drop on the original free-form task, while an 8:2 mixture recovered strong performance.The passage identifies output-format distribution as an explicit component of the training domain.Source: B. Target-Domain HiSpatial Adaptation · pp. 4–5

C. Qwen3-VL Fine-Tuning with Mixed-Domain Supervision

Qwen3-VL is adapted with mixed-domain instruction data and memory-efficient QLoRA for both model scales. Both adapted models outperform their corresponding pretrained baselines on the target benchmark.

C. Qwen3-VL Fine-Tuning with Mixed-Domain Supervision

  • Target spatial instruction data are combined with general multimodal instruction samples to adapt spatial tasks while retaining broader visual-language capabilities.The same mixed-domain strategy is used for both model families, though their optimization mechanisms differ.Source: C. Qwen3-VL Fine-Tuning with… · p. 5
  • 20.74% to 45.27% for Qwen3-VL-8B and 30.79% to 47.60% for Qwen3-VL-32B after mixed-domain QLoRA fine-tuning.The benchmark comparison is summarized in Table III.Source: C. Qwen3-VL Fine-Tuning with… · p. 5, Table III · p. 5
  • The Qwen3-VL branch covers instruction conversion, mixed-domain dataset construction, QLoRA optimization, checkpoint serialization and reloading, batch inference, and automated evaluation at both scales.The implementation completes the second model branch of the engineering pipeline.Source: C. Qwen3-VL Fine-Tuning with… · p. 5

D. Current Implementation Status

The implementation status covers completed HiSpatial and Qwen3-VL data, training, inference, and evaluation components. Reproducibility is supported by separated software environments, recoverable experiment artifacts, and a shared evaluation interface.

VI. Reproducibility and Software Organization

  • Separate environments isolate data conversion, training, inference, and evaluation dependencies, while a shared JSONL prediction schema enables model comparison without evaluator-specific changes.Source: §VI · pp. 5–6
  • Dataset JSONL conversion and image alignment are listed alongside bounding-box and styled-image input support.Source: §VI · p. 6

D. Current Implementation Status

  • The completed stack covers bounding-box/styled-image input support, MoGe point-map preprocessing, HiSpatial WebDataset construction and mixed-domain full fine-tuning plus checkpoint inference/evaluation, and Qwen3-VL instruction conversion and mixed-domain QLoRA fine-tuning.Source: §VI · p. 6

VI. Reproducibility and Software Organization

  • Experiments are designed to be recoverable from four artifacts: dataset JSONL, image root, model checkpoint and configuration, and exported evaluation results.Source: §VI · p. 6

VII. Discussion and Next Steps

The discussion reports strong HiSpatial results and gains from mixed-domain adaptation for both Qwen3-VL model sizes, with the approaches providing complementary adaptation strategies. The conclusion presents the pipelines as a reproducible foundation and identifies further comparisons and evaluations for future work.

VII. Discussion and Next Steps

  • Qwen3-VL-8B accuracy improved from 20.74% to 45.27%, and Qwen3-VL-32B accuracy improved from 30.79% to 47.60% under mixed-domain training.Source: §VII · p. 6, §VIII · p. 6
  • HiSpatial uses explicit 3D geometry, whereas Qwen3-VL gains through parameter-efficient multimodal instruction tuning.Source: §VII · p. 6
  • Future work will examine target/general data-mixture ratios, task-wise evaluation, input-format ablations, and monocular point maps versus geometry from 3D assets or camera metadata.Source: §VII · p. 6

VIII. Conclusion

  • HiSpatial overall accuracy rose from 1.73% to 79.89%, while accuracy on the separately collected styled split rose from 14.9% to 55.1%.Source: §VIII · p. 6
  • The pipelines provide a reproducible engineering foundation for further study of metric 3D reasoning, spatial grounding, multimodal data mixing, and geometry-aware adaptation.Source: §VIII · p. 6
Loading 2610.37216v1…