PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perception

Ziang Yan, Xinhao Li, Yinan He, Zhengrong Yue, Xiangyu Zeng, Yali Wang, Yu Qiao, Limin Wang, Yi Wang

arXiv:2509.21100v1cs.CV

TL;DR

MLLM reasoning is constrained by static visual perception, despite its importance for human-level multimodal understanding. VTTS introduces iterative perception with reinforcement learning and VTTS-80K, and VideoChat-R1.5 improves performance across more than 15 benchmarks. The paper concludes that iterative visual refinement supports more adaptive multimodal reasoning, while noting current visual-only scope and inference inefficiencies.

  • Problem

    Existing MLLM approaches mainly reason over visuals parsed once, limiting exploration of iterative visual perception for multimodal reasoning.

  • Method

    VTTS uses ITP to iteratively select and reprocess spatio-temporal regions, with reinforcement learning and VTTS-80K supporting visual dependency modeling.

  • Results

    Across more than 15 benchmarks, VTTS consistently improves MLLM performance, with average gains of 5.4% for Qwen2.5-VL-7B and 6.3% for Qwen2.5-VL-3B.

  • Takeaways & Limitations

    VTTS provides a test-time scaling framework that refines visual focus and supports more adaptive multimodal reasoning across video conversation, reasoning, and spatio-temporal perception tasks.

  • Takeaways & Limitations

    VTTS has been validated only for visual processing, and its iterative inference does not yet fully exploit visual-language priors or caching techniques.

Abstract

from arXiv · show

Inducing reasoning in multimodal large language models (MLLMs) is critical for achieving human-level perception and understanding. Existing methods mainly leverage LLM reasoning to analyze parsed visuals, often limited by static perception stages. This paper introduces Visual Test-Time Scaling (VTTS), a novel approach to enhance MLLMs' reasoning via iterative perception during inference. VTTS mimics humans' hierarchical attention by progressively refining focus on high-confidence spatio-temporal regions, guided by updated textual predictions. Specifically, VTTS employs an Iterative Perception (ITP) mechanism, incorporating reinforcement learning with spatio-temporal supervision to optimize reasoning. To support this paradigm, we also present VTTS-80K, a dataset tailored for iterative perception. These designs allows a MLLM to enhance its performance by increasing its perceptual compute. Extensive experiments validate VTTS's effectiveness and generalization across diverse tasks and benchmarks. Our newly introduced Videochat-R1.5 model has achieved remarkable improvements, with an average increase of over 5\%, compared to robust baselines such as Qwen2.5VL-3B and -7B, across more than 15 benchmarks that encompass video conversation, video reasoning, and spatio-temporal perception.

1 Introduction

VTTS addresses the limitation of one-time visual perception by enabling MLLMs to iteratively refine attention during inference. Its ITP mechanism, reinforcement learning, and VTTS-80K dataset support progressive spatio-temporal reasoning, with consistent gains across diverse benchmarks.

  • Motivation: Existing multimodal methods mainly deepen reasoning over parsed visuals or expand language-model search while retaining a single perception stage.These approaches include BoN, guided beam search, MCTS, and reinforcement learning with visual rewards.
  • VTTS: VTTS dynamically reallocates computation toward high-confidence spatio-temporal regions across iterations, guided by evolving textual predictions.The approach models dependencies between language and vision over time while progressively narrowing attention.
  • Iterative Perception: ITP predicts regions of interest from prior reasoning and reprocesses them for more detailed context, while reinforcement learning learns spatio-temporal focus policies.The objective extends autoregressive MLLM training with visual dependency modeling and uses GRPO with spatio-temporal annotations.
  • Dataset: VTTS-80K provides QA pairs with annotated spatio-temporal cues and chains of thought for learning critical visual segments.The dataset is designed to support iterative perception and reasoning through small-scale reinforcement fine-tuning.
  • Results: 5.4% average improvement for Qwen2.5-VL-7B and 6.3% for Qwen2.5-VL-3B across benchmarks spanning perception, video QA, and multimodal reasoning.The paper reports consistent increases across more than 15 benchmarks and MLLM variants.

2 Related work

Prior work improves visual tasks through specialized models, task-specific modules, or language-oriented test-time scaling, but these strategies can limit generalization or add complexity. VTTS is presented as an iterative visual-perception alternative, while Figure 3 depicts its inference process.

  • Multimodal Large Language Models: MLLMs have expanded from image captioning and visual question answering toward video understanding.Representative systems include BLIP-2, LLaVA, mPLUG-Owl, and later video-focused models.
  • Multimodal Video Understanding: Task-specialized strategies can achieve high accuracy but may compromise generalization across diverse scenarios.Videochat-TPO improves visual tasks and conversation through specialized modules, at the cost of additional complexity.
  • Test Time Scaling: LLM test-time scaling methods such as BoN, guided beam search, and MCTS increase inference computation, while MLLM scaling remains less thoroughly explored.Prior MLLM approaches primarily emulate LLM methods by extending outputs or related reasoning processes.
  • Reinforcement Learning: Reinforcement learning has improved LLM reasoning through verifiable rewards and has increasingly been extended to multimodal models.The related-work discussion situates multimodal reinforcement learning within broader advances in reasoning systems.
  • VTTS: VTTS implements coarse-to-fine visual processing by iteratively predicting high-confidence spatio-temporal regions and building dependencies between processed visuals.Figure 3 presents the framework’s iterative-perception inference procedure.

3 Methodology

VTTS extends multimodal reasoning from static visual processing to iterative perception, repeatedly narrowing attention to relevant spatio-temporal regions using evolving textual predictions. Its ITP mechanism combines ROI refinement, reinforcement learning with spatio-temporal supervision, and the VTTS-80K dataset to support adaptive reasoning at inference.

  • Visual Test-Time Scaling: VTTS simulates coarse-to-fine human attention by iteratively predicting high-confidence spatio-temporal regions and refining visual computation during inference.The process explicitly builds dependencies between processed visuals and evolving textual predictions.
  • Visual Dependency Modeling: VTTS extends autoregressive dependency modeling from language to visuals by generating a new focused visual input from the model’s previous answer estimate.The sampling function δ(Vk|Wa) produces Vk+1 with focus determined by the model’s kth estimation.
  • Reinforcement Fine-Tuning: Because standard visual-question-answer training lacks precise spatio-temporal descriptions, VTTS uses reinforcement learning to optimize ROI coordinates alongside answer and format outputs.The reward combines spatio-temporal alignment, answer accuracy, and output-format compliance; clue alignment uses IoU.
  • Reinforcement Fine-Tuning: Supervised fine-tuning alone hardly improves region discovery or subsequent reasoning, motivating reinforcement-based learning for iterative perception.The authors report this result from experiments optimizing the visual-language objective with textual and numerical regional descriptions.
  • Iterative Perception: ITP begins with broad processing, then reallocates computation toward identified cues while preserving global context.Videos use dense sampling in critical temporal segments and sparse sampling elsewhere; images combine focused regions with the full image.
  • Iterative Perception: The hierarchical processing improves feature extraction and reasoning about complex interactions between objects or events.This reported benefit follows the differential allocation of visual processing across focused and broader regions.
  • VTTS-80K Dataset: VTTS-80K provides QA pairs, relevant spatio-temporal segments, and reasoning traces, with verification and filtering before step-by-step reasoning annotations are generated.The pipeline validates QA consistency and cue relevance, creates Filtered QA, and then generates multimodal chain-of-thought explanations.

4 Experiments

Experiments evaluate VideoChat-R1.5 across video understanding, grounding, tracking, perception scaling, data ablations, and inference-time trade-offs. Results consistently support iterative perception and reinforcement-based training, with gains across diverse tasks and competitive efficiency.

  • General Evaluation: Experiments span video perception, general and long-video understanding, video knowledge modeling, grounded VideoQA, temporal and spatial grounding, and object tracking.The benchmark suite includes MVBench, PerceptionTest, VideoMME, MLVU, LVBench, LongVideoBench, VideoMMMU, NextGQA, ReXTime, RefCOCO variants, and GOT-10k.
  • General Evaluation: VideoChat-R1.5-7B and -3B achieve 70.6% and 68.1% on MVBench, outperforming the baseline by 2.2% and 1.6%, respectively.On VideoMME, the models surpass Qwen2.5-VL by 2% and 2.7%, while iterative perception exceeds single-step perception by 1.9% and 1.2%.
  • General Evaluation: The 7B model gains 6.6%, 0.7%, and 1.1% on LongVideoBench, LVBench, and MLVU without additional long-video training data.It also surpasses the baseline by 2.2% on VideoMMMU, while the 3B model improves by 1.5%.
  • Spatial-Temporal Tasks: VideoChat-R1.5 improves grounded VideoQA across QA and IoP metrics at both model scales, and its 7B variant surpasses GPT-4o across all ReXTime metrics.These evaluations test joint answer accuracy and localization of supporting temporal segments.
  • Spatial-Temporal Tasks: Fine-tuned VideoChat-R1.5-3B and -7B outperform all baselines for temporal grounding, with the 7B model reaching 71.6 on R@0.5.The model also reports substantial zero-shot improvements over previous baselines.
  • Ablation: Performance scales with perception iterations: VideoMME rises from 65.2 to 67.9, LongVideoBench from 61.4 to 62.9, and OCRBench from 856 to 866.The ablations report a clear scaling trend as the number of perception steps increases.
  • Ablation: Spatial and temporal grounding data improve corresponding fine-grained tasks, while chat data benefits long-video understanding, image perception, and spatial reasoning.The ablation isolates distinct contributions of VTTS-80K data components.
  • Ablation: VideoChat-R1.5-7B single inference reaches 67.2% on MMVet at 1.35 seconds and 65.2% on VideoMME at 13.21 seconds, balancing performance and latency.Multi-inference adds 1.1% performance at additional computational cost; reinforcement learning yields double-digit localization gains while SFT degrades backbone competence.

5 Conclusions

VTTS enhances multimodal reasoning through iterative visual perception and dynamic refinement of key spatio-temporal regions. Experiments report gains across more than 15 benchmarks, while the method remains limited to visual processing and lacks full inference optimization.

  • Conclusions: VTTS dynamically refines focus on key spatio-temporal regions to enhance multimodal reasoning and interpretability.The framework is supported by reinforcement learning tailored for multimodal contexts.
  • Conclusions: More than 15 benchmarks show VTTS outperforming strong baselines across video conversation, image reasoning, and spatio-temporal perception tasks.The reported results highlight iterative perception’s effectiveness for multimodal reasoning.
  • Conclusions: VTTS has been validated only for visual processing, while its inference is not fully optimized for visual-language priors or caching techniques.Extending the approach to additional modalities and improving engineering implementation remain future directions.

Appendix

The appendix expands dataset details, training and inference parameters, and case studies of model reasoning. It includes both successful examples and analyses of failure cases.

  • Appendix: The appendix provides additional details on the VTTS-80K dataset, including its composition and annotation types.These details support the paper’s main content.
  • Appendix: The appendix details the training and inference parameters used for the model.
  • Appendix: Case studies present successful reasoning examples alongside analyses of failure cases to characterize model performance.

6 Details of VTTS-80K

VTTS-80K combines several task components and heterogeneous annotations to support iterative perception across spatio-temporal localization and multimodal reasoning. Its training process adapts rewards to the annotations available in each data subset.

  • Dataset composition: VTTS-80K contains VideoQA, Temporal Clue, Image Reasoning, and Spatial Clue components curated from existing datasets.The dataset uses question, option, think, clue, and answer annotations.
  • Annotation types: Clue annotations encode either temporal information or spatial locations and relationships within the input.
  • Training adaptation: The training process dynamically selects reward functions according to the annotations available in each dataset subset.This aligns the reward mechanism with heterogeneous annotation coverage.
  • Annotation coverage: All VTTS-80K entries include Question and Think annotations, while other annotation types vary by data source.
  • Data sources: Grounding or tracking sources provide clue annotations, whereas QA sources focus on answering questions from visual and textual inputs.
  • Dataset scale: VTTS-80K comprises 15K temporal clues, 30K spatial clues, 80K Think annotations, and 50K QA pairs.This composition supports tasks ranging from fine-grained spatio-temporal localization to complex multimodal reasoning.

7 Training and Evaluation Details

The appendix reports separate settings for VTTS reinforcement-learning training and evaluation, including video-frame ranges, resolution constraints, and evaluation frame selection from temporal clues.

  • Training settings: VTTS reinforcement-learning training uses 4 to 768 video frames at 2 FPS with bounded pixel counts.The training configuration uses AdamW with a 2 × 10^-6 learning rate, zero weight decay, and batch size 16.
  • Input settings: Image and video inputs use a factor of 28 and shared pixel-count constraints across the reported training and evaluation settings.
  • Evaluation settings: VTTS evaluation uses 4 to 2048 video frames at 2 FPS and applies a 0.5 key ratio to select frames from time clues.

8 Case Study

Case studies show that iterative perception can recover critical visual clues after initial errors and support fine-grained spatio-temporal tasks with explanations. Failure analysis attributes remaining errors mainly to inaccurate localization and loss of global context.

  • Case Studies: Iterative perception corrected initially incorrect answers in video QA and image reasoning by identifying critical clues across multiple reasoning rounds.The multi-step process integrates insights from multiple perception iterations.
  • Case Studies: The model performed grounded video QA, temporal grounding, grounded image QA, spatial grounding, and tracking while generating explanations for its decisions.The cases emphasize both accurate spatial or temporal outputs and reasoned explanations.
  • Failure Cases: Failures primarily arose from inaccurate localization of critical clues, difficulty finding evidence for knowledge-based questions, and loss of global context during local focus.These issues expose a tension between fine-grained localization and holistic understanding.

9 Discussions

The discussion presents VTTS as a useful approach for multimodal reasoning while noting deployment, safety, societal, privacy, and bias concerns. The accompanying cases illustrate iterative perception and its successes and failures across visual tasks.

  • Limitations: The method may increase inference time because each additional perception step requires further computation, challenging real-time or latency-constrained applications.The limitation is presented alongside improved performance from iteration.
  • Limitations: Intermediate reasoning steps may generate unintended or inappropriate content that propagates into final outputs, creating safety concerns requiring safeguards.The concern arises from the “thinking” process inherent in iterative perception.
  • Broader Impacts: Potential societal risks include misinformation, deepfakes, hallucinations, privacy misuse, and amplified bias in applications such as news, science, law, hiring, policing, and healthcare.The discussion connects these risks to enhanced content manipulation, detailed visual processing, and improved reasoning capabilities.
  • Safeguards: The paper follows the NeurIPS Code of Ethics, but outlines no specific technical safeguards against deepfake generation for VTTS.Detection tools and responsible release strategies are identified as broader mitigation pathways.
  • Case Studies: Iterative perception is illustrated for image, grounded video QA, temporal grounding, grounded image QA, detection, tracking, and failure cases.The cited figures cover both successful applications and failures in image and video iteration perception.
Loading 2509.21100v1…