PaperlayerView exact revision on arXiv

Source-linked AI summary

DynThinker: Unifying Static 3D and Dynamic 4D Spatial Reasoning

Qihang Cao, Haoyuan Li, Zihan Guo, Sensen Gao, Zihui Gao, Huixin Zhang, Fan He, Jianhua Han, Hang Xu, Xiaodan Liang, Jia-Wang Bian

arXiv:2610.20598v1cs.CV

TL;DR

Existing models can struggle to excel at both static 3D and dynamic 4D spatial reasoning, and post-training can trade one capability against the other. DynThinker combines group-wise parameter averaging with semantic-guided fusion of spatial and temporal priors. Across ten metrics, it achieves the highest average score among evaluated models and strong, balanced 3D and 4D performance.

  • Problem

    Existing models can struggle to excel at both static 3D and dynamic 4D reasoning, while 3D-focused post-training can weaken pretrained dynamic competence.

    Source: §1 · p. 2

  • Method

    DynThinker combines configurable group-wise averaging of pretrained parameters with semantic-guided integration of spatial and temporal priors into intermediate VLM representations.

    Source: §1 · p. 2

  • Results

    Across ten metrics, DynThinker achieves the highest average score at 60.5, surpassing GPT-5.6 Sol at 59.5, with state-of-the-art results among compared open-source models on ReVSI, DSR-Bench, and DSI-Bench.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The results show that strong static 3D and dynamic 4D capabilities can be unified within one model.

    Source: §6 · p. 10

    The method is a research prototype and should not be directly deployed in safety-critical settings without additional validation and safeguards.

    Source: ETHICS STATEMENT · p. 11

Abstract

from arXiv · show

Achieving general spatial intelligence requires a unified understanding of both static 3D and dynamic 4D. However, we find that existing models struggle to excel at both: 3D post-training can weaken dynamic understanding, while 4D-specialized models often lack strong static 3D reasoning. Our analysis reveals that this spatial--dynamic interference is highly non-uniform across parameter groups, with the balance between static and dynamic capabilities particularly sensitive to changes in late-layer MLPs. To address this interference, we propose DynThinker, a unified framework for jointly strengthening static and dynamic spatial reasoning. Decoupled Exponential Moving Average (D-EMA) applies configurable group-wise averaging to constrain drift in interference-sensitive pretrained parameters while preserving flexibility for capability enhancement. Spatial-Dynamic Fusion (SDF) complements D-EMA by selectively integrating spatial and temporal priors from a dynamic geometry encoder into intermediate VLM representations under semantic guidance. Across ten metrics spanning nine static 3D and dynamic 4D benchmarks, DynThinker achieves the highest average score among all evaluated models. Notably, DynThinker achieves 56.0 on ReVSI, 67.2 on DSR-Bench, and 51.2 and 40.1 under the sample-wise and group-wise evaluation protocols of DSI-Bench, respectively, setting new state-of-the-art results among the compared open-source models across all four evaluations within a unified model.

1 INTRODUCTION

The introduction frames unified spatial reasoning as a challenge because strengthening static 3D reasoning can degrade dynamic 4D competence. DynThinker pairs parameter preservation with semantic-guided fusion and reports strong, balanced results across both settings.

1 INTRODUCTION

  • 3D-focused post-training can improve static reasoning while degrading pretrained dynamic competence, complicating efforts to build models strong in both.Existing preservation strategies may require auxiliary replay data, careful data mixing, constrained adaptation, or additional regularization objectives.Source: §1 · p. 2
  • The static–dynamic tension is uneven across parameters: changes to late-layer MLPs are particularly sensitive, and restoring them recovers dynamic performance while partly reversing 3D gains.This opposing response identifies late-layer MLPs as a particularly pronounced site of interference.Source: §1 · p. 2
  • DynThinker combines D-EMA’s configurable group-wise averaging of sensitive pretrained parameters with SDF’s semantic-guided integration of spatial and temporal priors into intermediate VLM representations.The framework is intended to preserve flexibility for capability enhancement while strengthening static geometry and temporal understanding.Source: §1 · p. 2
  • Across ten metrics, DynThinker scores 60.5 on average, surpassing GPT-5.6 Sol at 59.5 and achieving state-of-the-art results among compared open-source models on ReVSI, DSR-Bench, and DSI-Bench.Ablations support complementary roles for D-EMA and SDF in mitigating interference while strengthening spatial reasoning.Source: §1 · pp. 2–3

2 RELATED WORK

Related work covers benchmarks for static and dynamic spatial reasoning, methods for integrating geometry into VLMs, and approaches to preserving capabilities during post-training. DynThinker builds on these directions by combining spatial and dynamic priors in one VLM and applying group-specific averaging.

2.1 STATIC AND DYNAMIC SPATIAL INTELLIGENCE

  • Static and dynamic benchmarks cover distinct settings, including multi-view 3D layout, video motion and temporal continuity, observer-versus-object motion, evolving 3D relations, and stereo-derived scenarios.The paper uses these complementary targets to motivate joint reasoning across static 3D and dynamic 4D.Source: §2.1 · p. 3

2.2 GEOMETRY INTEGRATION IN VISION–LANGUAGE MODELS

  • Geometry reaches VLMs through feature fusion, distillation, or intermediate-layer integration; SDF fuses spatial and dynamic priors within a single VLM, adaptively balancing their contributions.Related approaches include geometry encoders for static or dynamic scenes, while GeoSR uses separate models for static and dynamic reasoning.Source: §2.2 · p. 3

2.3 CAPABILITY PRESERVATION DURING POST-TRAINING

  • Prior preservation methods include checkpoint interpolation and within-run averaging, while D-EMA targets uneven spatial capability interference with configurable group-wise averaging horizons normalized by training duration.The passage notes that standard EMA hyperparameters have also been adapted to training scale, such as batch size.Source: §2.3 · p. 3

3 DIAGNOSING SPATIAL–DYNAMIC INTERFERENCE

The analysis finds that 3D specialization can trade dynamic performance for static gains, with the interference concentrated particularly in late-layer MLPs. These findings motivate selective temporal consolidation, which the passage reports remains effective as training data scales.

Static–dynamic trade-off

  • VSI-Bench improves while VLM4D declines along the interpolation toward the 3D-specialized checkpoint, revealing a static–dynamic trade-off.Static performance begins to saturate at intermediate interpolation coefficients while dynamic performance continues declining toward the specialized checkpoint.Source: §3 · p. 4, Figure 2 · p. 4

Parameter-group sensitivity

  • Restoring late-layer MLPs toward pretrained values recovers more dynamic performance than restoring other groups, while partially reversing static 3D gains.The opposing response indicates particularly pronounced interference in late-layer MLPs and motivates stronger preservation for sensitive parameters.Source: §3 · p. 4

Implications for capability preservation

  • D-EMA applies configurable group-wise temporal averaging to consolidate sensitive pretrained parameters while leaving online adaptation unconstrained.Unlike hard freezing, temporal averaging retains influence from earlier parameter states during training.Source: §3 · pp. 4–5
  • SDF integrates complementary spatial and temporal priors from a dynamic geometry encoder into intermediate VLM representations under semantic guidance.Importance gating selectively modulates both feature streams using the VLM’s semantic context.Source: Figure 3 · p. 5
  • As post-training data scales, D-EMA preserves dynamic competence while static 3D performance continues benefiting; without it, both capabilities degrade with more data.The figure caption describes D-EMA as improving both static and dynamic performance across the MLLM4D training-data scale.Source: Figure 2 · p. 4, §3 · p. 5

4 METHOD

DynThinker combines semantic-guided fusion of spatial and temporal priors with group-specific temporal averaging to jointly enhance spatial reasoning and preserve capabilities during adaptation.

4 METHOD

  • DynThinker combines SDF for joint capability enhancement with D-EMA for interference-aware capability preservation.SDF injects spatial and temporal priors into intermediate VLM representations under semantic guidance; D-EMA averages pretrained parameters by group.Source: §4 · p. 5

4.1 MODEL ARCHITECTURE

  • The architecture uses a dynamic geometry encoder to provide complementary spatial and temporal features aligned with intermediate VLM representations.It densely encodes all input frames but decodes only a temporally ordered subset to reduce sequential decoder cost.Source: §4.1 · pp. 5–6

4.2 SPATIAL-DYNAMIC FUSION

  • SDF uses semantic representations to gate spatial and dynamic priors separately, then combines the aligned streams through residual scaling.The residual gate starts at zero, and the priors are injected into existing visual representations without adding LLM tokens.Source: §4.2 · pp. 6–7

4.3 DECOUPLED EXPONENTIAL MOVING AVERAGE

  • D-EMA averages pretrained parameter groups with configurable horizons while leaving online optimization unconstrained and excluding newly introduced parameters from averaging.The sensitive late-MLP group can receive stronger averaging; decay conversion across schedules preserves the initial weights’ end-of-training contribution.Source: §4.3 · pp. 7–8

5 EXPERIMENT

DynThinker is evaluated across static 3D and dynamic 4D benchmarks, with component ablations and scaling experiments examining its performance and design. It achieves the strongest reported cross-benchmark balance among the compared models.

5 EXPERIMENT

  • Evaluation spans seven spatial-static benchmarks (ReVSI, VSI-Bench, MMSI-Bench, MindCube-Tiny, ViewSpatial, SITE, CV-Bench) and three temporal-dynamics benchmarks (VLM4D, DSR-Bench, DSI-Bench), with sample-wise and group-wise DSI-Bench protocols.The training mixture contains approximately 2.68M samples.Source: §5.1 · p. 8, §5 · p. 8

5.1 EVALUATION RESULTS

  • 60.3 average: DynThinker outperforms all compared open-source models, improves GeoThinker by 11.2 points, and surpasses GPT-5.6 Sol by 0.8 points.It improves all four dynamic metrics while retaining strong static 3D performance.Source: §5.1 · p. 8
  • Under the same 128-frame budget, DynThinker surpasses Cambrian-P by 4.0 percentage points on ReVSI and slightly improves on GeoThinker's 55.9%.Gains include object and room size estimation, relative distance, and relative direction.Source: §5.1 · p. 9
  • Disabling inference-time token compression raises average score from 60.3 to 60.5, but VSI-Bench falls from 69.3 to 67.8.VLM4D rises from 60.0 to 62.3, and both DSI-Bench protocols improve; retaining more visual tokens benefits some tasks but not overall performance consistently.Source: §5.1 · p. 9

5.2 ABLATION STUDY

  • D-EMA raises VSI-Bench from 51.31 to 53.14 and VLM4D from 45.20 to 53.57 in the 4B ablation.Adding VDPM and spatial and temporal priors further improves both capabilities; the full 4B model reaches 56.56 on VSI-Bench and 59.74 on VLM4D, and scaling to 8B raises these to 58.16 and 61.78.Source: §5.2 · p. 9, Table 3 · p. 9
  • 66.74 on full VLM4D: the 1.5M-sample DynThinker model exceeds MLLM-4D RFT at 61.0.On VLM4D Real, 800K samples reach 63.82, above MLLM-4D's reported 59.7 with 2M SFT samples and 63.1 with RFT.Source: §5.2 · p. 9, Table 3 · p. 9

6 CONCLUSION

DynThinker combines D-EMA and SDF to unify static 3D and dynamic 4D reasoning, with the conclusion reporting strong, balanced performance across benchmarks. The supplementary material outlines additional analyses and implementation details.

6 CONCLUSION

  • DynThinker uses group-wise temporal averaging to preserve pretrained capabilities and integrates spatial and temporal priors into VLM representations under semantic guidance.The conclusion reports strong and balanced 3D and 4D performance across diverse benchmarks.Source: §6 · pp. 9–10
  • The authors report using generative AI for language editing and other research support, while stating that the authors determined the core hypothesis, methods, model design, and scientific findings.They state that AI-assisted code and technical and numerical claims were manually reviewed or checked.Source: AI USE STATEMENT · p. 11

A APPENDIX/SUPPLEMENTAL MATERIAL

  • The appendix lists further analyses of MLP boundaries, weight interpolation, downstream applications, frame and scaling ablations, and spatial and temporal reasoning.It also includes results on embodied manipulation and sensitivity analysis.Source: A APPENDIX/SUPPLEMENTAL MATERIAL · p. 17

B.1 MODEL CONFIGURATIONS

  • For 32-frame training, the VDPM encoder processes all frames while the sequential decoder uses eight uniformly sampled frames and interpolates features across time.Both streams are resampled to the VLM visual-token grid before fusion.Source: §B.1 · p. 17
  • Visual token compression uses merge size 4 for inputs longer than eight frames and the backbone's native size 2 for inputs of at most eight frames.This reduces token sequence length for longer inputs while retaining a finer spatial grid for shorter ones.Source: §B.1 · p. 17

B.2 DATA CURATION

The scaled training mixture combines spatial reasoning, temporal dynamics, and general video data.

B.2 DATA CURATION

  • The 2.68M-sample mixture includes spatial instruction data, temporal-dynamics data, and general video samples.The listed sources include Cambrian-S and VLM-3R spatial instructions, DSR and MLLM4D training data, and LLaVA-Hound and Cambrian-S-3M video samples.Source: §B.2 · pp. 17–18

C OVERVIEW OF ADDITIONAL RESULTS

The appendix describes additional experiments across scaled, controlled spatial, dynamic, and embodied settings. These settings vary data, frame counts, backbones, or policy integration to examine different aspects of DynThinker.

Main scaled regime

  • The main scaled regime uses Qwen3-VL-8B, 32 input frames, and about 2.68M samples, with cross-benchmark results and further analyses in the appendices.Additional evaluations include STI-Bench and qualitative spatial and dynamic comparisons.Source: C OVERVIEW OF ADDITIONAL RESULTS · p. 18

Dedicated 4D data-scaling setting

  • The dedicated 4D scaling setting varies MLLM4D training data on Qwen3-VL-8B, matching DynThinker and MLLM4D backbones and samples at the same budgets.This setting is separate from the main 2.68M-sample mixture.Source: C OVERVIEW OF ADDITIONAL RESULTS · p. 18

Static 3D vanilla regime

  • Controlled static spatial experiments use eight input frames and 234K SPAR-7M plus 64K LLaVA-Hound samples to study components from GeoThinker.The appendix also examines D-EMA configurations starting from direct spatial supervised fine-tuning, a different starting model.Source: C OVERVIEW OF ADDITIONAL RESULTS · p. 18

Dynamic 4D vanilla regime

  • Controlled dynamic preservation experiments add 200K MLLM4D samples to the 298K-sample static mixture and use eight input frames.The appendix compares preservation strategies and studies inference-time frame count and token compression.Source: C OVERVIEW OF ADDITIONAL RESULTS · p. 18

Embodied setting

  • The embodied setting incorporates SDF into StarVLA-OFT's Qwen3-VL-4B backbone and trains one policy across four LIBERO suites for 30,000 steps.Setup and results are reported in Appendix I.1.Source: C OVERVIEW OF ADDITIONAL RESULTS · p. 18

D.1 FURTHER FRAME ABLATION

The ablations examine frame scaling, training-mixture and drift-budget choices, benchmark comparisons, and qualitative examples of spatial and dynamic reasoning. Across these settings, DynThinker shows stronger dynamic performance while retaining competitive spatial reasoning, with results varying by task and configuration.

D.1 FURTHER FRAME ABLATION

  • 56.0 on ReVSI-Bench at 128 evaluation frames, up from 50.5 at 16 frames; DynThinker surpasses GeoThinker from 64 frames onward.GeoThinker remains stronger on VSI-Bench and VSI-Debiased, so the advantage is concentrated in ReVSI-Bench.Source: Table 4 · p. 18, §D.1 · p. 19

D.2 SCALING ABLATION STUDY

  • 59.87 average follows expanding the MLLM4D subset from 10% to 50%, and adding 430K general video samples produces the final 60.29 average.The larger drift budget improves average scores but trades off capabilities; under 32-frame training, VLM4D falls from 61.3 to 57.0 as the budget rises from 25% to 50%.Source: §D.2 · p. 19, Table 5 · p. 19

D.3 EVALUATION ON STI-BENCH

  • 40.1 overall at 16 evaluation frames places DynThinker fourth among the compared STI-Bench configurations, with the highest reported 3D video grounding score of 41.9.At 30 frames, its overall score is 39.9 versus GeoThinker’s 27.6.Source: §D.3 · p. 19, Table 6 · p. 20

D.4 COMPARISON TO GEOTHINKER

  • DynThinker improves broadly over GeoThinker on dynamic, video, and embodied benchmarks, while its static spatial results are mixed and MVBench is a notable exception.The comparison identifies DSR, DSI, and VLM4D among the particularly large gains.Source: §D.4 · p. 19, Figure 4 · p. 20

D.5 VISUALIZATION

  • Across real and synthetic videos, DynThinker rejects false premises and answers motion or rotation questions that the pretrained or spatially specialized models miss.The examples include preserving correct pretrained judgments on vehicle motion and movement away from the camera, while correcting errors on stationary-object and perspective-dependent rotation questions.Source: Figure 5 · p. 21, Figure 6 · p. 21, Figure 7 · p. 22

E.1 FURTHER VLM4D COMPARISON

  • Under deterministic VLM4D evaluation, the final DynThinker model scores 60.07, rising to 62.27 without token compression; the scaled-data model reaches 66.74, 5.74 points above the best officially reported overall score.Rescoring released MLLM4D responses yields comparable or generally no higher scores than the official results, supporting the authors’ claim that the gain is not an answer-extraction artifact.Source: Table 7 · p. 23, §E.1 · p. 24

F.1.2 VANILLA REGIME

  • 58.1 on VSI-Bench with Qwen3-VL-8B is the highest overall score in the vanilla-regime comparison, while the 4B variant scores 56.5.The corresponding improvements over GeoThinker are 5.7 and 5.2 points, respectively.Source: §F.1.2 · pp. 24–25, Table 8 · p. 24

F.2.2 D-EMA DRIFT ALLOCATION

  • 58.16 on VSI-Bench and 61.78 on VLM4D make a shared 12.5% drift budget the best evaluated allocation for Qwen3-VL-8B; Qwen2.5-VL-7B instead uses 12.5% for MLPs in blocks j > 4 and 100% for other pretrained parameters.For Qwen2.5-VL-7B, this asymmetric allocation improves VSI-Bench to 50.37 while retaining 48.56 on VLM4D.Source: §F.2.2 · p. 25, §F.2.1 · p. 25

F.2.3 TRAINING AND COMPONENT ABLATIONS.

These ablations compare D-EMA and SDF, preservation strategies, inference-time frame scaling, and the MLP boundary used for selective consolidation. Results show that the preferred balance depends on backbone, task, and evaluation setting.

F.2.3 TRAINING AND COMPONENT ABLATIONS.

  • On Qwen3-VL-4B, D-EMA recovers VSI-Bench and VLM4D scores after Static 3D SFT, and adding SDF raises them further to 56.56 and 59.74.On Qwen3-VL-8B, SDF raises VLM4D from 59.74 to 61.78 while maintaining Static 3D performance; D-EMA also stabilizes pretrained representations during joint adaptation.Source: §F.2.3 · p. 25, Table 10 · p. 26

G.1.1 SETTING

  • The controlled temporal-dynamic vanilla regime uses 8 input frames, a spatial mixture of sampled SPAR-7M and LLaVA-Hound data, and 10% of MLLM4D training data.Evaluation covers VSI-Bench and VLM4D and compares capability-preservation strategies and inference-time frame scaling.Source: §G.1.1 · p. 27

G.1.2 PRESERVATION STRATEGIES

  • 55.31 on VSI-Bench and 60.57 on VLM4D make D-EMA the best-balanced compared strategy for Qwen3-VL-4B, exceeding pretrained scores on both benchmarks.Conventional EMA scores 52.55 and 55.12, while the pretrained VLM4D score is 57.10.Source: §G.1.2 · p. 27, Table 11 · p. 27

G.1.3 INFERENCE-TIME FRAME SCALING

  • 64.42 VLM4D accuracy is the 8B model’s peak at 16 frames, while VSI-Bench rises from 49.93 at 8 frames to 58.93 at 64 frames without token compression.With VDPM decoding capped at 8 sampled frames, 128-frame inputs achieve 57.97 on VSI-Bench and 62.44 on VLM4D without token compression.Source: §G.1.3 · p. 28, Table 12 · p. 28

H.1 SELECTION OF THE MLP LAYER BOUNDARY

  • Restoring MLP layers j > 4 recovers nearly the same dynamic performance as restoring all MLPs, with little change in spatial accuracy; the authors therefore designate those layers as interference-sensitive.The observed trade-off is that retaining more specialized MLP layers favors spatial reasoning at the expense of dynamic recovery.Source: §H.1 · p. 28

H.2.1 INTERPOLATION ON GEOTHINKER

Interpolation analyses show that late-layer MLPs are especially sensitive to spatial–dynamic interference: restoring them recovers dynamic performance, sometimes while retaining strong static performance. The results support selectively consolidating these parameters with D-EMA.

H.2.2 INTERPOLATION ON DYNTHINKER WITHOUT D-EMA

  • Without D-EMA, spatial specialization raises VSI-Bench accuracy from 54.90% to 71.06% but lowers VLM4D accuracy from 57.10% to 50.33%.Thus, integrating spatial and dynamic geometry features alone does not prevent interference during adaptation.Source: §H.2.2 · p. 29
  • 58.31% VLM4D accuracy after restoring late-layer MLPs in DynThinker without D-EMA exceeds both the specialized checkpoint at 50.33% and pretrained backbone at 57.10%.Retaining 50% of specialized late-layer MLP weights yields 53.13% VLM4D accuracy while VSI-Bench remains 70.89%, close to the specialized checkpoint's 71.06%.Source: §H.2.2 · p. 29

I.1.1 SETTING

  • StarVLA-OFT is the embodied baseline, an MLP-head VLA architecture decoding continuous actions; the evaluation trains 30,000 steps with batch size 16 and learning rate 2.5 × 10−5.The protocol follows StarVLA-OFT on LIBERO.Source: §I.1.1 · pp. 29–30

H.2.1 INTERPOLATION ON GEOTHINKER

  • Restoring late MLP layers j > 4 raises VLM4D accuracy to 51.15%, nearly matching all-MLP restoration at 51.59% and exceeding attention-only restoration at 46.86%.Restoring early MLP layers j ≤ 4 yields 43.61%; these findings support boundary b = 4 and stronger temporal consolidation for later MLPs in D-EMA.Source: §H.2.1 · p. 29, Figure 9 · p. 29, Table 13 · p. 30
  • 50% specialized-weight retention recovers VLM4D accuracy from 42.40 to 49.17 while maintaining a VSI-Bench score of 70.12.The analysis describes this as a global acquisition–retention trade-off.Source: §H.2.1 · pp. 28–29, Table 13 · p. 30

I.1.2 EVALUATION RESULTS.

On LIBERO, DynThinker achieves the highest observed average success rate with one policy across four suites. It also improves over StarVLA-OFT and GeoThinker under the same backbone and training schedule.

I.1.2 EVALUATION RESULTS.

  • 97.3% average success rate across all four LIBERO suites is the highest observed, with one policy; under the same Qwen3-VL-4B backbone and 30K-step schedule, it exceeds StarVLA-OFT by 0.7 and GeoThinker by 0.6 percentage points.It also improves GeoThinker's Long-suite result from 91.2% to 92.8%.Source: §I.1.2 · pp. 30–31, Table 15 · p. 32

Related papers

Loading 2610.20598v1…