PaperlayerView exact revision on arXiv

Source-linked AI summary

ChronoVision: Temporal Reasoning via Latent State Reconstruction

Yifan Shen, Jian Xu, Boyi Li, Yuner Zhang, Tianjiao Yu, Bingxuan Li, Houze Yang, Rushi Wang, Xu Cao

arXiv:2608.05631v1cs.CV

TL;DR

Multimodal large language models struggle with complex visual temporal reasoning, motivating ChronoVision’s latent-state reconstruction framework and strict image-ordering benchmark. ChronoVision achieves state-of-the-art results, including 74.8% in-domain accuracy on Vbvr-VQA.

  • Problem

    Multimodal large language models remain limited on complex visual reasoning, while language-based benchmarks can permit shortcuts instead of genuine temporal understanding.

  • Method

    ChronoVision reconstructs transformed visual states in latent space, locates dynamic regions, and uses strict image ordering to evaluate temporal progression.

  • Results

    ChronoVision achieves state-of-the-art performance across Vbvr-VQA and IntPhys2, including 74.8% accuracy on Vbvr-VQA’s In-Domain split.

  • Takeaways & Limitations

    The results support latent visual reconstruction and process grounding as an effective framework for visual temporal reasoning.

  • Takeaways & Limitations

    The study validates the approach mainly in a moderate-scale 9B model and relies on dense auxiliary supervision, limiting established scalability and adaptability.

Abstract

from arXiv · show

Multimodal large language models excel at passive perception but struggle with complex visual cognitive tasks requiring multi-step temporal reasoning. This degradation largely stems from the inherent ambiguity of language-based reasoning, which often fails to accurately articulate continuous visual transformations. To address this, we propose ChronoVision, a multimodal framework designed to align visual logic with latent imagery. During supervised fine-tuning, a Reconstructive Visual Head predicts the latent representation of the final transformed state, while an ROI Attention Locating module focuses the model on key visual evidence via semantic span queries. In post-training, we apply reinforcement learning with an implicit process grounding mechanism, guided by a composite reward function that evaluates outcome correctness, latent process alignment, and unsupervised visual focus. Furthermore, we introduce Vbvr-VQA, a novel dataset that evaluates temporal tracking by reformulating video reasoning into a strict image-ordering task. Experiments demonstrate that ChronoVision achieves state-of-the-art performance on Vbvr-VQA with 74.8% in-domain and 71.6% out-of-domain accuracy, alongside a strong 55.0% accuracy on IntPhys2, a highly challenging cross-domain benchmark.

1 Introduction

ChronoVision addresses multimodal models’ difficulty with continuous visual and temporal reasoning by combining latent visual-state reconstruction, focused attention, and implicit process grounding. It is evaluated with Vbvr-VQA, an image-ordering benchmark designed to reduce linguistic shortcuts.

  • Motivation: Multimodal large language models excel at open-vocabulary perception but remain limited on spatial intelligence, visual reasoning, visual memory, and multi-step temporal reasoning.They often recognize static content rather than model continuous visual states with real physical properties.
  • Motivation: Text-based Chain of Thought creates a bottleneck because language cannot precisely preserve critical spatial information in continuous visual transformations.This is especially problematic for complex three-dimensional rotations and continuous physical trajectories.
  • Vbvr-VQA: Vbvr-VQA reformulates video reasoning as ordering six shuffled frames chronologically, forcing models to understand temporal evolution rather than exploit linguistic shortcuts.The task provides an initial frame and text prompt, then requires arranging candidate frames into the correct sequence.
  • ChronoVision: ChronoVision uses a Reconstructive Visual Head to predict the final transformed state in latent space and an ROI Attention Locating module to focus on dynamic key regions.The reconstructive module uses scattered candidate-frame information, while semantic locating cues guide fine-grained attention.
  • Training: Reinforcement learning with implicit process grounding evaluates outcome correctness, latent visual alignment, and attention focus to optimize reasoning trajectories and improve long-horizon robustness.These criteria are combined in a Composite Reward Function.
  • Results: Experiments report state-of-the-art performance on Vbvr-VQA and the best overall accuracy on the challenging cross-domain IntPhys2 benchmark.The introduction presents these results as evidence supporting ChronoVision’s effectiveness.

2 Dataset and Benchmark

Vbvr-VQA addresses the lack of benchmarks for abstract visual reasoning by reformulating video reasoning as text-free chronological ordering. It combines diverse physical scenarios with dense annotations identifying transformation regions and their spatial locations.

  • Dataset Motivation: Vbvr-VQA targets abstract visual reasoning, including multistep transformation inference, causal temporal dynamics, and complex scene understanding, which existing benchmarks largely lack.The dataset is built upon the open-source Very Big Video Reasoning dataset.
  • Benchmark Design: The benchmark enforces text-free strict ordering to eliminate linguistic shortcuts and require visual comprehension.Unlike multiple-choice or free-form formats, the task prevents exploiting question stems or answer options through linguistic pattern-matching.
  • Benchmark Design: 100 distinct task generators span fluid dynamics, kinematic collisions, and continuous spatial rotations under one constrained evaluation interface.The unified format tests temporal and physical causal reasoning across heterogeneous domains.
  • Task Construction: The model receives an initial frame and prompt, then predicts the exact chronological permutation of six shuffled candidate images.The six candidates represent evolutionary states sampled by uniformly dividing the ground-truth video into six temporal intervals.
  • Annotations: Dense annotations synthesize intermediate reasoning traces and <LOCATE>...</LOCATE> cues identifying dynamic transformation regions.Precise bounding boxes provide spatial supervision for downstream attention modules and mitigate grounding-perception mismatch.

3 Methodology

ChronoVision models chronological permutations from a query image, shuffled candidates, and a task prompt using a unified visual-textual backbone. Its methodology combines latent final-state reconstruction, localized attention alignment, and reinforcement learning with implicit process grounding.

  • Overall Framework: The model predicts a chronological permutation from a query image, six shuffled candidate images, and a textual reasoning prompt.A visual encoder produces dense visual tokens, while the language-model backbone autoregressively generates the output sequence.
  • Reconstructive Visual Head: The Reconstructive Visual Head predicts the latent representation of the final chronological image from shuffled candidate-frame hidden states.It is trained with Mean Squared Error alongside autoregressive cross-entropy, linking sequence prediction to the final-state visual representation.
  • ROI Attention Locating: The ROI Attention Locating module generates semantic locate cues and concentrates intermediate-layer attention on annotated transformation regions.Its Attention Condensation loss aligns text-to-image attention with the ground-truth bounding box, and the combined objective is Ltotal = LSFT + αLAC.
  • Reinforcement Learning: ChronoVision uses GRPO with three reward components bounded within [0, 1]: final outcome, latent-grounding process, and unsupervised visual focus.These rewards assess exact sequence matching, sentence-level alignment between reasoning intents and candidate visual features, and concentration of visual self-attention.
  • Reinforcement Learning: The total reward is Rtotal = ω1Rout + ω2Rlatent + ω3Rfocus, and policy optimization uses group-relative advantages with a clipped surrogate objective and KL penalty.Balancing weights combine the three rewards during GRPO optimization.

4 Experiments

ChronoVision achieves strong temporal-ordering performance on Vbvr-VQA while preserving general vision-language capabilities. Ablations and out-of-domain evaluations show benefits from latent-state reconstruction, ROI attention, composite rewards, and physical reasoning generalization.

  • Evaluation Protocol: Exact Match accuracy requires the entire generated sequence of six candidate frames to exactly match the ground-truth chronological order.Vbvr-VQA evaluates fine-grained visual understanding and logical reasoning in visual temporal transformations.
  • Vbvr-VQA Results: 74.8% In-Domain, 71.6% Out-of-Domain, and 73.2% overall accuracy establish ChronoVision as state-of-the-art on Vbvr-VQA.It surpasses Claude Opus 4.6 and larger models including Qwen 3.5 397B and GPT o3.
  • General Vision-Language Capabilities: 78.8 vs. 78.4 on MMMU and 85.9 vs. 85.7 on MathVista show performance comparable to vanilla Qwen 3.5 9B across seven general multimodal benchmarks.The results indicate that ChronoVision’s attention locating and reconstructive objectives do not degrade general visual-language understanding.
  • Training-Process Ablation: +3.2% ID and +2.8% OOD improvements result from adding the Reconstructive Visual Head, while ROI attention adds +1.2% overall, from 69.0% to 70.2%.The RVH explicitly predicts latent visual states, and the ROI module filters spatial semantic noise.
  • Out-of-Domain Generalization: 55.0% overall accuracy on IntPhys 2 yields a +6.5% absolute gain over Qwen 3.5 9B (48.5%), while Easy reaches 62.5% with a +11.5% gain.ChronoVision also obtains overall scores of 45.89 on Video-Holmes and 74.7 on LongVideo-Reason.

5 Related Works

Section 5 situates ChronoVision within four related research areas: vision-language models for vision reasoning, visual cognition, ROI selection and cropping, and reinforcement learning.

  • 5 Related Works: The work is closely related to VLMs for vision reasoning, visual cognition, ROI selection and cropping, and reinforcement learning.A comprehensive discussion of these areas is provided in Appendix F.

6 Conclusion

ChronoVision addresses the text bottleneck in long-horizon abstract visual reasoning through implicit visual reasoning, fine-grained spatial focus, and reinforcement learning to mitigate textual chain-of-thought errors.

  • ChronoVision is introduced to resolve the text bottleneck in long-horizon abstract visual reasoning.
  • The framework integrates a Reconstructive Visual Head and Region of Interest Attention module during supervised fine-tuning.These components provide implicit visual reasoning capabilities and fine-grained spatial focus.
  • A reinforcement learning stage uses a composite reward function to mitigate compounding errors in textual chains of thought.

Limitation · A Additional information of the Dataset

ChronoVision’s current validation centers on latent sequence reconstruction and process grounding in a moderate-scale 9B multimodal language model. The study identifies scalability and supervision density as areas for future investigation.

  • Limitation: ChronoVision is primarily validated within a moderate-scale 9B MLLM setting.The validation focuses on latent sequence reconstruction and process grounding.
  • Limitation: Future work could examine scalability to larger multimodal backbones.
  • Limitation: Future work could also evaluate larger and more diverse training corpora.
  • Limitation: The framework relies on dense auxiliary supervision.The passage identifies semantic locating cues and spatial bounding-box annotations as examples.
  • Limitation: The study reports strong temporal reasoning and visual cognition capabilities on Vbvr-VQA and IntPhys2.

A.1 Statistics of the Dataset … A.4 Data Quality Assurance and Human Validation

Vbvr-VQA reformulates temporal video reasoning as six-frame chronological ordering, with large-scale training, in-domain and zero-shot out-of-domain testing, and broad cognitive coverage. Its sequence labels derive from source-video time, while annotations and manual reviews provide supervision and quality assurance.

  • A.1 Statistics of the Dataset: 1,000,000 samples are used for ChronoVision’s Supervised Fine-Tuning and Reinforcement Learning stages.
  • A.1 Statistics of the Dataset: 500 testing samples comprise 250 in-domain examples and 250 zero-shot out-of-domain examples with unseen task generators and extreme physical scenarios.
  • A.1 Statistics of the Dataset: 6! = 720 candidate-frame permutations make random exact ordering probability 1/720 ≈0.14%, while accuracy exceeding 70 percent across 500 instances supports non-random temporal reasoning.
  • A.2 Visual Examples of Dataset: The benchmark spans five cognitive categories: Fluid Intelligence, Crystallized Intelligence, Mental Simulation, Visuospatial Cognition, and Transformation.
  • A.2 Visual Examples of Dataset: Videos are uniformly divided into six temporal intervals; the final frame from each interval is extracted and randomly shuffled as candidates labeled A through F.
  • A.3 Annotation Details of Dataset: Each inference query combines ordering instructions with the original video description, and presents the initial frame, prompt, and six shuffled candidate frames.
  • A.3 Annotation Details of Dataset: GPT-5 generates dense spatio-temporal annotations with bounding boxes and verb-centric <LOCATE>...</LOCATE> cues to supervise ROI Attention Locating.
  • A.4 Data Quality Assurance and Human Validation: The chronological sequence remains absolute ground truth from the source video’s continuous temporal axis, independently of GPT-5-generated text.

B More Implementation Details … C.7 Ablation Study on Linear Probing

ChronoVision’s implementation uses a two-stage SFT–RL pipeline with attention-based latent grounding, while ablations examine evaluation strictness, latent-state evolution, reasoning dependence, causal intervention, and linear-probe predictiveness. The analyses support intermediate latent sequences as actively guiding and causally sustaining temporal reasoning.

  • B.1 Training Paradigm and Hyperparameters: Training uses SFT followed by GRPO-based RL on 8 NVIDIA H100 GPUs with DeepSpeed ZeRO-3 optimization.The pipeline is explicitly divided into sequential supervised fine-tuning and reinforcement-learning stages.
  • B.1 Training Paradigm and Hyperparameters: SFT uses AdamW with a 2×10−5 peak learning rate, batch size 128, weight decay 0.05, 3 epochs, cosine decay, and 3% warmup; RL uses 1×10−6 learning rate and G = 8.RL additionally sets ϵ = 0.2, a KL penalty of 0.01, and reward weights ω1 = 1.0, ω2 = 0.5, and ω3 = 0.1.
  • B.2 Justification for Attention Alignment over Bounding Box Regression: Attention alignment supervises internal distributions rather than coordinate regression, preserving dense visual features for RVH-based latent-state prediction.This choice targets complex, non-rigid transformations and avoids the information bottleneck of four discrete coordinates.
  • B.3 Selection of the Designated Attention Layer: A single intermediate attention layer is selected before fine-tuning by held-out validation, because multi-layer aggregation dilutes localization signals and adds semantic noise.The selected layer is the one with the highest average concentration on target boxes across the base backbone’s layers.
  • C.1 Ablation Study on Evaluation with Partial Match Metrics: Kendall’s τ complements Exact Match by measuring pairwise ordering consistency for N = 6 frames, and ChronoVision outperforms four prominent baselines across ten cognitive categories.Exact Match remains the strict criterion because reversing even two frames breaks the causal chain, whereas Kendall’s τ isolates minor errors.
  • C.2 Ablation Study on Latent Sequence Evolution: For correct trajectories, latent-sequence MSE decreases across reasoning steps, indicating stepwise convergence toward the ground-truth final state.The comparison uses the RVH’s predicted final-state representation against the frozen visual encoder’s ground-truth embedding, averaged over 10 trajectories.
  • C.3 Ablation Study on Chain-of-Thought; C.4 Ablation Study on Reasoning Dependence: Removing intermediate chain-of-thought causes a performance drop, while incorrect reasoning prefixes produce much smaller degradation for full ChronoVision than for the SFT w/o head variant.These ablations test whether textual reasoning guides latent reconstruction and whether RVH passively follows the textual chain of thought.
  • C.5 Ablation Study on Intermediate State Perturbation; C.6 Ablation Study on Latent Sequence Intervention; C.7 Ablation Study on Linear Probing: Perturbing an intermediate latent state increases downstream latent error without reconvergence, while latent patching and linear probing test whether these states causally influence and predict the final permutation.The linear probe freezes ChronoVision and predicts exact image order from intermediate latent sequences at different reasoning steps.

C.8 Comparison with Related Temporal Reasoning Methods · D Additional Evaluation on Real-World Video Reasoning · D.1 Evaluation on Video-Holmes

ChronoVision outperforms related temporal-reasoning methods under matched backbone and data conditions, with results supporting explicit final-state supervision and direct supervision of visual transformations. Its temporal reasoning transfers beyond Vbvr-VQA to real-world video reasoning, achieving an overall Video-Holmes score of 45.89.

  • C.8 Comparison with Related Temporal Reasoning Methods: ChronoVision is compared with R1-VL, VL-Cogito, and Latent Sketchpad using the same Qwen3.5-9B backbone and identical supervised fine-tuning data.R1-VL and VL-Cogito supervise reasoning without an explicit final visual-state target, while Latent Sketchpad’s visual latents are not tied to the final ordering state.
  • C.8 Comparison with Related Temporal Reasoning Methods: ChronoVision obtains the highest overall, in-domain, and out-of-domain accuracy among the compared temporal-reasoning methods.The comparison suggests that directly supervising the final visual state benefits temporal ordering.
  • C.8 Comparison with Related Temporal Reasoning Methods: Forcing direct one-step final-sequence generation consistently reduces performance across In-Domain Vbvr-VQA categories.The CoT ablation demonstrates that step-by-step textual reasoning guides latent sequence generation.
  • C.8 Comparison with Related Temporal Reasoning Methods: The full ChronoVision model degrades less than SFT without the visual head when given an incorrect reasoning prefix.This result indicates that ROI plus reinforcement learning helps internalize visually grounded constraints against incorrect intermediate textual logic.
  • C.8 Comparison with Related Temporal Reasoning Methods: Gaussian noise injected at Step 30 disrupts a previously successful latent reasoning trajectory, which diverges and fails to reconverge to the clean path.Intermediate latent sequences therefore causally drive and sustain accurate reasoning, according to the perturbation study.
  • D Additional Evaluation on Real-World Video Reasoning: ChronoVision is additionally evaluated on real-world video reasoning to test transfer beyond the Vbvr-VQA ordering setting.The evaluation uniformly samples 32 frames from each video.
  • D.1 Evaluation on Video-Holmes: 45.89 overall score is achieved by ChronoVision on Video-Holmes, compared with 42.00 for GPT-4o and 45.00 for Gemini-2.5-Pro.Results cover seven reasoning categories and indicate transfer to real-world video reasoning tasks.

D.2 Evaluation on LongVideo-Reason

ChronoVision is evaluated on LongVideo-Reason, a long-video benchmark covering temporal, goal, plot, and spatial reasoning. It achieves an overall score of 74.7 and the highest score in all four reported categories.

  • Evaluation on LongVideo-Reason: LongVideo-Reason measures temporal, goal, plot, and spatial reasoning over long videos.The benchmark is cited as Chen et al., 2026.
  • Evaluation on LongVideo-Reason: 74.7 overall score is achieved by ChronoVision on LongVideo-Reason.LongVideo-Reason measures reasoning over long videos.
  • Evaluation on LongVideo-Reason: ChronoVision obtains the highest score in each of LongVideo-Reason’s four reported categories.The categories are temporal, goal, plot, and spatial reasoning.

E Visualization Example … G.2 Human Validation Subjects and Instructions

ChronoVision addresses temporal visual reasoning through latent-state reconstruction, ROI-focused evidence selection, and reinforcement learning, with qualitative examples showing recovery of continuous transformations. The paper also situates these methods among related work and documents dataset licensing and human validation procedures.

  • E Visualization Example: ChronoVision reconstructs temporal order by tracking continuous visual transformations, including animal faces becoming size-sorted along a baseline and a camera orbit reversing an 8-block sculpture’s orientation.The examples emphasize positional changes, viewpoint-induced structural changes, foreshortening, side-face visibility, and the gradual emergence of the opposite side.
  • F Related Works; F.1 VLMs for Vision Reasoning: The related-work discussion identifies a shift from static visual reasoning toward continuous visual and latent-space reasoning, while positioning image ordering and final-state latent supervision as ChronoVision’s temporal solution.It also describes visual instruction tuning and reinforcement fine-tuning as approaches for aligning multimodal perception with reasoning.
  • F.2 Visual Cognition: ChronoVision targets visual-cognition deficits by training models to internally simulate the latent visual state of the correct final outcome.The surrounding discussion contrasts these deficits with human cognitive development and notes failures in object permanence, spatial dynamics, and physical consistency.
  • F.3 ROI Selection and Cropping: ROI research progresses from tool-based cropping and explicit coordinate prediction toward attention-driven localization based on internal “where-to-look” signals.The passage notes that attention-based signals can remain sensitive to layer-wise attention behavior.
  • F.4 Reinforcement Learning: Reinforcement-learning research addresses optimization stability, sampling efficiency, credit assignment, and biases in GRPO, whereas prior methods mainly optimize textual reasoning.Examples include DAPO, CPPO, VinePPO, PPO, GRPO, and Dr. GRPO.
  • G Ethics Statement and Reproducibility; G.1 Artifact License and Data Usage: The Vbvr-VQA benchmark is derived from the open-source Very Big Video Reasoning dataset, whose Apache License 2.0 permits academic and commercial use, modification, and distribution.The paper states that the benchmark’s research focus and applications are essentially consistent with the source dataset’s intended use.
  • G.2 Human Validation Subjects and Instructions: Human validation covered the entire 500-sample test set and a random 500-sample training subset to verify physical correctness of ground-truth sequences and GPT-5 reasoning traces.Reviewers were graduate-level researchers specializing in computer vision and physics-based machine learning, participating voluntarily without direct monetary compensation.
  • G.2 Human Validation Subjects and Instructions: Reviewers mentally simulated each transformation, checked physical plausibility and rationale accuracy, and produced region-grounded explanations of the specified chronological sequence under a strict verification protocol.The protocol was designed to exclude synthetic hallucinations and enforce adherence to verifiable physical laws.
Loading 2608.05631v1…