PaperlayerView exact revision on arXiv

Source-linked AI summary

DK-VLM: Geometry as Position for Spatial Reasoning in Vision-Language Models

Han Su, Meng Tian, Tianyu Huang, Mingyu Gao, Lennox Wei, Dongxu Li, Jianhua Han, Hang Xu, Xiaohe Wu, Wangmeng Zuo

arXiv:2610.22356v1cs.CV

TL;DR

Vision-language models remain unreliable at spatial judgments across views and as observers move, while integrating reconstruction geometry with pretrained semantic representations is challenging under limited spatial supervision. DK-VLM combines native and geometry-based attention and adds supervision for reference-frame updates across steps. It achieves the highest six-benchmark average among compared methods and the highest overall scores on VSI-Bench and ReVSI-32.

  • Problem

    VLMs remain unreliable at estimating spatial relations across views and updating position and heading as observers move, while fusing reconstruction features requires aligning representations under limited spatial supervision.

    Source: §1 · pp. 1–2

  • Method

    DK-VLM retains native attention in a semantic kernel, adds a geometric kernel using estimated 3D points and camera poses, and trains stepwise reference-frame updates with turn and heading targets.

    Source: §1 · pp. 2–3

  • Results

    DK-VLM-8B has the highest average across six spatial benchmarks and the highest overall scores among compared methods on VSI-Bench (74.33) and ReVSI-32 (58.28).

    Source: §1 · p. 3, §5 · p. 12

  • Takeaways & Limitations

    The reported gains support combining geometric relation modeling with stepwise supervision, while the full model preserves general video understanding.

    Source: §1 · p. 3, §5 · p. 12

Abstract

from arXiv · show

Vision-language models still struggle to understand 3D spatial relations across views and update spatial judgments as the observer moves. A common approach fuses 3D reconstruction features into visual tokens, but aligning these features with pretrained semantic representations remains challenging under limited spatial supervision. We propose DK-VLM with Dual-Kernel Attention, which computes attention from both a visual token's input position and its geometric position in the scene. A semantic kernel retains the pretrained model's native attention and positional encoding, while a separate geometric kernel uses estimated 3D points and camera poses to directly modulate attention through point displacements and relative view transformations. Then, a learned gate fuses their outputs. Geometry thus enters attention as position rather than as fused reconstruction features. We further introduce Reference-Frame Composition (RC) supervision to train the model to update its reference frame across successive judgments. It provides stepwise turn and heading targets derived from verified single-step annotations. Among the compared methods, DK-VLM-8B achieves the highest average score across six spatial benchmarks and the highest overall scores on VSI-Bench (74.33) and ReVSI-32 (58.28). RC supervision further improves multi-step spatial reasoning, while the full model preserves general video understanding. Project page: https://sumuru789.github.io/DK-VLM-website/.

1 Introduction

DK-VLM addresses spatial reasoning across views by introducing scene geometry into attention while retaining the pretrained model’s native positional encoding. It also supervises reference-frame updates across steps; DK-VLM-8B leads the reported six-benchmark average and two benchmark overall scores.

1 Introduction

  • VLMs remain unreliable at estimating distance, direction, camera motion, and cross-view correspondence, especially for tasks chaining multiple judgments.Spatial reasoning requires tracking 3D relations across views and updating position and heading as the observer moves.Source: §1 · p. 1
  • Fusing reconstruction features into visual tokens requires aligning representations learned for reconstruction with pretrained semantics, a challenge under limited spatial supervision.The passage says this adaptation may limit effective use of geometry and interfere with existing semantic representations.Source: §1 · pp. 1–2
  • Dual-Kernel Attention retains native attention and positional encoding while a separate geometric kernel uses 3D points and camera poses to modulate token relations, with a learned gate fusing outputs.Geometry enters attention as position rather than as reconstruction features fused into token content.Source: §1 · pp. 2–3
  • Reference-Frame Composition supplies stepwise turn and heading targets to supervise how spatial judgments update their reference frame across successive steps.The targets are derived by connecting verified single-step relations that share landmarks or views within a scene.Source: §1 · p. 3
  • DK-VLM-8B achieves the highest average across six spatial benchmarks and the highest overall scores on VSI-Bench (74.33) and ReVSI-32 (58.28) among compared methods.The model is also reported to preserve general video understanding.Source: §1 · p. 3

2 Related Work

Related work introduces spatial reasoning through training data and geometric representations, and uses geometry to shape attention. DK-VLM instead modulates attention with estimated scene geometry while retaining native positional encoding.

2 Related Work

  • Spatial training approaches include instruction tuning, progressive curricula, verifiable rewards, and deterministic geometric environments, while compositional supervision for reference-frame changes has received less attention.The cited examples include VST, Cambrian-S, SpatialLadder, SpaceR, and SpatialEvo (Yang et al., 2025b,c, Li et al., 2025b, Ouyang et al., 2025, Li et al., 2026a).Source: §2 · p. 3
  • Geometry-augmented VLMs encode coordinates or add reconstruction features to visual tokens, whereas DK-VLM uses estimated 3D points and camera poses to modulate attention directly.The passage contrasts this with coordinate embeddings and reconstruction-feature fusion (Zheng et al., 2024, Zhu et al., 2024, Liang et al., 2026, Zheng et al., 2025, Fan et al., 2026, Li et al., 2026b).Source: §2 · p. 4
  • Geometric attention methods use camera transformations, 3D points, rays, or viewpoints to parameterize token relations, while retaining a pretrained VLM’s native positional encoding remains less explored.The cited methods include GTA, PRoPE, RayRoPE, and ViewRoPE (Miyato et al., 2024, Li et al., 2025c, Wu et al., 2026, Xiang et al., 2026).Source: §2 · p. 4

3 Method

DK-VLM combines backbone-native semantic attention with geometry-conditioned attention, and trains both spatial judgments and reference-frame updates. Its method also supervises pairwise geometric representations during joint training.

3.1 Dual-Kernel Attention

  • At selected layers, DK-VLM retains native semantic attention and adds a geometry-conditioned kernel over valid visual tokens, then gates both outputs into the residual stream.The semantic kernel retains native Q/K/V projections and mRoPE; the geometric kernel uses full attention among geometry-valid visual tokens.Source: §3.1 · pp. 4–5, Figure 2 · p. 5

3.2 Reference-Frame Composition

  • RC links compatible verified relations along landmark or view paths and recomputes stepwise turn targets, using each heading as the next reference.Targets are derived from source annotations, and cumulative headings are used when available.Source: §3.2 · p. 7

3.3 Joint Training Objective

  • The training objective jointly optimizes instruction SFT and pairwise geometric prediction from ordered token pairs, targeting log-distance and direction.The prediction head is used during training and removed for inference.Source: §3.3 · p. 8

4 Experiments

Experiments evaluate DK-VLM’s spatial reasoning, reference-frame supervision, general video understanding, and model components. The reported results show strong spatial benchmark performance, improvements from RC supervision, and comparable general video performance.

4.1 Experimental Setup

  • The experiments use Qwen2.5-VL-7B and Qwen3-VL-8B, evaluate seven spatial and two general-video benchmarks, and train on combined data for one epoch.The training data includes approximately 1.6M constructed RC samples, of which 150K two- and three-step chains are used after filtering and sampling.Source: §4.1 · pp. 8–9

4.2 Spatial Reasoning

  • 74.33 on VSI-Bench and 58.28 on ReVSI-32 are the highest overall scores among compared methods.Both DK-VLM variants outperform corresponding GeoThinker models on VSI-Bench; DK-VLM-8B also outperforms GeoThinker-8B on route planning in both benchmarks.Source: Table 1 · p. 9, §4.2 · p. 9
  • 64.09 is DK-VLM-8B’s highest average score across six spatial benchmarks, and it ranks first on four of them.The comparison includes spatial post-training and geometry-enhanced methods.Source: Table 2 · p. 10, §4.2 · p. 10

4.3 Effect of RC Supervision

  • RC improves route planning on VSI-Bench and ReVSI-32, multi-step reasoning on MMSI-Bench, and person-perspective tasks in ViewSpatial.The comparison uses Qwen3-VL-8B models with and without RC supervision; the ViewSpatial result covers three person-perspective tasks.Source: Table 3 · p. 10, §4.3 · p. 10
  • RC improves Progress and Chain Acc. at every evaluated chain length, including four-step chains excluded from RC training.Progress measures correct steps before the first error; Chain Acc. measures full-sequence accuracy.Source: Table 4 · p. 11, §4.3 · p. 11

4.4 General Video Understanding

  • DK-VLM-8B scores 1.63 points higher on VideoMME and 0.49 points lower on MVBench than the Qwen3-VL-8B fine-tuned baseline.The authors characterize these results as comparable general video performance.Source: Table 5 · p. 11, §4.4 · p. 11

4.5 Ablation Studies

  • Point GTA and Camera GTA together outperform either alone; replacing both with identity transforms lowers the score by 4.31 percentage points.The geometric branch and its parameters remain in place for the identity-transform comparison, and pairwise geometric supervision improves spatial judgments.Source: §4.5 · p. 11, Table 6 · p. 11
  • RC yields a larger gain on DK-VLM than on Qwen3-VL-8B trained on the same instruction data, supporting complementarity with geometric relation modeling.Both models are compared with and without RC supervision.Source: §4.5 · p. 12

5 Conclusion

DK-VLM introduces geometry as position in attention and adds RC supervision for multi-step spatial reasoning. The conclusion reports the highest average score across six spatial benchmarks while preserving general video understanding.

5 Conclusion

  • DK-VLM-8B achieves the highest average score across six spatial benchmarks while preserving general video understanding.The conclusion attributes further multi-step spatial reasoning improvements to Reference-Frame Composition supervision and describes geometric relation modeling and stepwise supervision as complementary.Source: §5 · p. 12

AI Use Statement

The authors report using AI assistants for manuscript drafting, translation, polishing, literature retrieval, and code implementation, and state that they reviewed the assisted work and take responsibility for its accuracy and integrity.

AI Use Statement

  • The authors used AI assistants for manuscript work, literature retrieval, and code implementation, then reviewed the assisted content and code.They state that they take full responsibility for the work’s accuracy and integrity.Source: AI Use Statement · p. 12

A.1 Geometry Preprocessing

Geometry preprocessing maps each merged visual token to a 3D point and associates tokens with frame-level camera poses. The resulting points and camera translations are normalized in a shared coordinate representation for geometric attention.

A.1 Geometry Preprocessing

  • Each merged visual token’s image-region center is mapped to the dense 3D point map, with preprocessing adjustments for resizing, cropping, and padding.Low-confidence points below the within-video 20th-percentile threshold of finite depth confidence are filtered; token coordinates are bilinearly interpolated from retained neighbors.Source: §A.1 · p. 17
  • Points are expressed in the first camera’s coordinate frame, then centered and scaled together with camera translations using the same scale factor.All visual tokens from a frame share its camera pose, and this common representation supports different geometry frontends with the same attention computation.Source: §A.1 · p. 17

A.2 Derivation of the Point GTA Relative Transform

The Point GTA transform represents pairwise geometric relations using relative view rotation and scaled point displacement. Its geometric attention head allocates separate channel subspaces to point and camera relations.

A.2 Derivation of the Point GTA Relative Transform

  • Each geometric attention head has 128 channels, including 48 for Point GTA and 24 for Camera GTA.The remaining 56 channels are separate; point and camera subspaces contain 12 and 6 four-dimensional blocks, respectively.Source: §A.2 · p. 17
  • Point GTA uses geometrically spaced displacement scales from 0.25 to 4.0 across its 12 blocks.The scales follow λb = 0.25 · 16^(b/11) for b = 0, . . . , 11.Source: §A.2 · p. 17
  • The complete pairwise relation is formed by multiplying the transform of token i with the transform of token j.The described relation includes relative source-view rotation and scaled point displacement.Source: §A.2 · pp. 17–18

A.3 Pairwise Loss Reduction and Boundary Cases

The pairwise loss trains predictions of geometric distance and direction from sampled ordered token pairs. Sampling and validity rules define which pairs contribute to each term and to the minibatch average.

A.3 Pairwise Loss Reduction and Boundary Cases

  • The prediction head maps pair features to log-distance and three-dimensional direction predictions, with the direction L2-normalized.It uses LayerNorm and a two-layer GELU MLP; hidden dimensions are 3584 for 7B and 4096 for 8B.Source: §A.3 · p. 18
  • Each sample draws 256 ordered token pairs, targeting 50% cross-frame pairs for multi-frame inputs.Candidates are sampled separately by distance tertiles; single-frame inputs use within-frame pairs only, and unused quotas transfer between groups.Source: §A.3 · p. 18
  • Samples with valid pairs contribute equally to the pairwise minibatch average.The loss also handles the case where the minibatch has no valid pairs.Source: §A.3 · p. 18
  • Pairs with dij ≤ ε, including zero-distance pairs, remain in the distance term but are excluded from the direction denominator; when Vn = ∅, direction loss is zero.Samples with Pn = ∅ do not enter the pairwise batch average; when B+ = ∅, batch pairwise loss is zero. Otherwise, samples with valid pairs contribute equally to Lpair.Source: §A.3 · p. 18

A.4 Reference-Frame Composition Details

Reference-Frame Composition constructs multi-step turn and heading targets from verified single-step annotations. Its procedures normalize and check source relations, then form chains within individual scenes or media items.

A.4 Reference-Frame Composition Details

  • RC verification checks source-angle consistency and algebraic consistency without using images, XYZ coordinates, or ground-truth 3D states.An independent validator recomputes derived targets and checks scene, split, and source provenance.Source: §A.4 · p. 19
  • VSI-RC normalizes source angles to clockwise-positive degrees and checks agreement with the source direction label.Inverse relations and angle closure are checked with a 2° circular tolerance.Source: §A.4 · p. 19
  • VSI-RC connects verified relations into landmark paths within one media item, rejecting paths with repeated landmarks.After the initial turn, later targets correct the source facing convention by 180° to use the traversal-centered reference.Source: §A.4 · p. 19
  • VSI-RC creates discrete-action targets only when every turn falls within conservative straight, right, back, or left angle bins.Its prompt supplies the initial reference point, bearing target, and ordered waypoints while withholding the derived turn sequence.Source: §A.4 · p. 19
  • SPAR-RC extracts yaw from structured camera-motion annotations, converts it to clockwise-positive angles, and excludes pitch and translation.Repeated directed view pairs are deduplicated; accepted relations form chains without repeated views, and the first view is the heading origin.Source: §A.4 · p. 20
  • RC produces four target types: VSI turn angles and eligible discrete actions, plus SPAR local yaw increments and cumulative headings.Each target sequence is serialized as a JSON array in a separate training sample, and the validator checks its provenance against source relations.Source: §A.4 · p. 20

A.5 Output Projection and Gate Initialization

The appendices specify DK-VLM’s projection and gate initialization, training and evaluation setup, and sensitivity analyses. Ablations report the effects of injection depth, placement, geometry frontends, frame counts, and alternative geometry-injection strategies.

A.5 Output Projection and Gate Initialization

  • DK-VLM initializes the shared output projection from the native projection and sets the gate logit to zero for an initial equal kernel mix.Each selected layer learns its own mixing ratio.Source: §A.5 · p. 20

A.6 Implementation Details and Parameter Counts

  • Training uses approximately 1.8M instruction samples spanning spatial reasoning and general video data.Both variants train for one epoch on 32 NVIDIA H200 GPUs; training takes approximately 11 days for 7B and 10 days for 8B.Source: §A.6 · p. 20, §A.7 · p. 21

A.7 Training Data Composition

  • The final RC training set has 150K entries and includes only two- and three-step chains.The separate evaluation set covers two-, three-, and four-step chains; RC training uses only two- and three-step chains.Source: §A.7 · p. 21, §A.8 · p. 21

A.8 RC Chain Evaluation

  • RC evaluation scores numeric turns within 10° and exact-match discrete actions, reporting Progress and Chain Acc. by chain length.Progress credits only consecutive correct predictions from the first step; Chain Acc. requires every step to be correct.Source: §A.8 · p. 22

A.9 Injection Layer Configuration

  • Distributed five-layer injection outperforms early and late placement by 0.62 and 1.76 percentage points, respectively.Increasing injection layers from three to five improves VSI-Bench by 0.77 points, while five to seven adds 0.06; five layers are the default.Source: §A.9 · p. 23, Table 11 · p. 23, Table 10 · p. 23

A.10 Geometry Frontends and Input Frame Counts

  • 74.33 is the VSI-Bench score with VGGT-Ω, the highest among tested geometry frontends at 32 frames without retraining.Other frontend scores range from 72.99 to 73.91, 0.42–1.34 points below the default.Source: §A.10 · p. 24, Table 12 · p. 24
  • 75.32 is the highest VSI-Bench score at 128 frames, while 32 frames approaches it with fewer inputs.Increasing from 8 to 32 frames adds 9.04 points, while 32 to 128 adds 0.99.Source: §A.10 · p. 24, Table 13 · p. 24

A.12 Comparison of Geometry Injection Strategies

  • 62.78 is DK-VLM’s VSI-Bench score without pairwise geometric loss, exceeding Qwen SFT by 3.30 points and five-layer embedding addition by 3.04.Single-kernel GTA scores 45.98 with three layers and 39.14 with five; five-layer DK-VLM exceeds it by 23.64 points.Source: §A.12 · p. 25, Table 14 · p. 25
Loading 2610.22356v1…