FrameMind: Frame-Interleaved Video Reasoning via Reinforcement Learning

1 INTRODUCTION

• Fixed sampling commits to one strategy regardless of question needs, risking insufficient temporal scope for event questions or inadequate spatial detail for localized analysis.Broad coverage suits finding when an event occurs, while localized questions may require high-resolution frames.Source: §1 · p. 1, Figure 1 · p. 2
• FiCOT lets models pause textual reasoning, identify knowledge gaps, and request targeted visual evidence during inference.FrameMind extracts high-resolution frames at specific timestamps or samples sequences from targeted intervals.Source: §1 · p. 2
• DRFS trains adaptive sampling policies across a resolution ladder spanning low-resolution temporal scanning and high-resolution spatial focus.The paper presents DRFS as a way to expose models to diverse temporal-spatial trade-offs during learning.Source: §1 · p. 2
• DRFS-GRPO learns sampling strategies from trajectory-level rewards without requiring expensive frame-level supervision.The paper describes it as an efficient reinforcement-learning algorithm for complex perception-reasoning policies.Source: §1 · p. 2
• FrameMind significantly outperforms existing open-source models and achieves performance competitive with top proprietary systems across MVBench, MLVU, and VideoMME.The introduction reports validation on a diverse suite of challenging video-understanding benchmarks.Source: §1 · p. 2

2.1 REASONING IN MULTIMODAL LLMS

• Multimodal RL work injects perceptual evidence into reasoning, yet perceptual granularity often remains fixed and hour-level video bandwidth allocation remains unresolved (OpenAI et al., 2024; DeepSeek-AI et al., 2025; Chen et al., 2025).Related approaches include visual operators, representation-level focusing, and structured supervision signals.Source: §2.1 · pp. 2–3

2.2 VIDEO UNDERSTANDING WITH MLLMS

• Video systems balance temporal coverage and spatial fidelity through compression, sparsification, adaptive scheduling, and longer-context modeling, but many retain static presets or heuristic schedules (Gao et al., 2023a; Yu et al., 2023; Shen et al., 2024; Ye et al., 2025).Benchmarking increasingly reports accuracy alongside alignment and budget metrics for equal-budget comparisons.Source: §2.2 · p. 3

2.3 TOOL-AUGMENTED MLLMS

• Tool-augmented reasoning offers plan–act–reflect routines and video grounding scaffolds, but autonomous choices about when, what, and how to invoke perception tools remain insufficiently addressed (Shen, 2024; Gao et al., 2023b; Wu et al., 2025).The stated gap concerns compute-constrained reasoning over frames and mixed video lengths.Source: §2.3 · p. 3

3.1 FRAMEWORK OVERVIEW

• FrameMind replaces fixed presampled-frame processing with an agent that requests visual information during reasoning based on its understanding and identified knowledge gaps.The framework overview presents this as an iterative perception-reasoning loop.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 FRAME-INTERLEAVED CHAIN-OF-THOUGHT (FICOT)

• FiCOT interleaves textual reasoning, tool calls, and returned visual evidence across turns, with FrameAt retrieving a high-resolution frame and VideoClip sampling a targeted interval.Training uses video-question-answer triplets and outcome-based rewards rather than expensive frame-level annotations; execution stops at an answer or after three turns.Source: §3.2 · pp. 4–5

3.3 DYNAMIC RESOLUTION FRAME SAMPLING (DRFS)

• DRFS generates parallel visual inputs across a resolution ladder, from many small frames for temporal scanning to fewer large frames for spatial focus.The method trains across this spectrum so the policy can learn which sampling strategy suits different videos and questions.Source: §3.3 · p. 5

3.4 REINFORCEMENT LEARNING WITH DRFS-GRPO

• DRFS-GRPO compares parallel rollout rewards with the group average, reinforcing above-average trajectories and suppressing below-average ones in a PPO-style update.The algorithm is designed to use the group structure generated by DRFS for policy learning.Source: §3.4 · pp. 5–6

3.5 REWARD DESIGN

• The reward combines task success and valid output with tool-use and efficiency shaping, including a bonus for finding answers in two or three turns.Tool-use reward is gated by answer correctness, with a small exploration reward and a larger reward when tool use leads to a correct answer.Source: §3.5 · p. 6

4.1 EXPERIMENTAL DETAILS

• Training uses a three-turn protocol, 32–64 DRFS frames spanning 224×224 to 448×448, and targeted clips of 8–20 frames at 448×448.The base policy is Qwen2.5-VL-7B; experiments run on one server with 8×NVIDIA A100-80GB GPUs.Source: §4.1 · p. 7

4.2 TRAINING DATA FOR FRAMEMIND

• The ∼7.6K training instances cover fine-grained perception, narratives, spatio-temporal relations, physical reasoning, and event explanations; DRFS data pairs long-form global scans with shorter precise-focus clips.The data is intended to elicit active visual probing, including coarse localization with VideoClip and high-resolution inspection with FrameAt.Source: §4.2 · p. 7

4.3 BENCHMARKS

• Benchmarks span short-video temporal perception (MVBench), long-video comprehension and localization (MLVU), and multimodal reasoning across durations from short clips to hour-long videos (Video-MME).The benchmark passages describe complementary duration and reasoning-task coverage.Source: §4.3 · p. 7

4.5 MAIN RESULTS

• 64.2% MVBench, 48.6% MLVU, and 60.9% VideoMME; FrameMind beats LongVA on MLVU using up to 4× fewer frames and surpasses GPT-4V on VideoMME.The authors report state-of-the-art performance among open-source peers under comparable configurations.Source: Table 1 · p. 8, §4.5 · pp. 8–9

4.6 ABLATION STUDY

• With the exploration bonus, tool reward and accuracy rise earlier and reach higher plateaus; without it, learning is slower and plateaus lower.The tested bonus is an unconditional 20% tool-use bonus; removing it leaves tool reward zero unless the final answer is correct.Source: Figure 3 · p. 9, §4.6 · p. 9

5 CONCLUSION

• FrameMind learns to choose between broad temporal scans and high-resolution focus, and the authors report state-of-the-art results as a step toward more flexible and efficient video understanding.The conclusion describes FiCOT as active evidence gathering during reasoning and DRFS-GRPO as the end-to-end training algorithm.Source: §5 · p. 10

FRAMEMIND: FRAME-INTERLEAVED CHAIN-OF-THOUGHT FOR VIDEO REASONING VIA REINFORCEMENT LEARNING

• The supplementary document adds method details and experimental results to the paper.The passage introduces an outline but does not provide its contents.Source: APPENDIX · p. 16

B.1 DETAILED REWARD FUNCTION SETUP

• A 0.5 turn-efficiency bonus applies when the final response contains 2–3 closed turn-summary blocks.The reward is intended to favor concise multi-turn reasoning.Source: §B.1 · p. 16

• The agent’s toolbox is designed for targeted visual evidence gathering.Source: §B.2 · p. 17
• The section introduces the implementation of the visual tools.Source: §B.2 · p. 17
• The section also introduces the tools’ parsing mechanism.Source: §B.2 · p. 17

• FrameAt extracts the closest frame to a specified timestamp and resizes it to 448 × 448 for detailed inspection.It is used to zoom in on a precise moment.Source: §B.3 · p. 17
• The model invokes tools through a special <tool_call> tag, which a regular-expression parser uses to extract the function call.Source: §B.3 · p. 17
• Malformed calls, invalid parameters, or out-of-range timestamps return an error message that the agent can use to correct its next action.Source: §B.3 · p. 17

C BENCHMARK DETAILS

• MLVU tests long-video reasoning and localized-detail recall, Video-MME assesses perceptual and cognitive abilities across modalities, and MVBench targets fine-grained temporal understanding in short clips.Source: C BENCHMARK DETAILS · pp. 17–18

D BASELINES DETAILS

• Standard video MLLM baselines are primarily supervised-finetuned models, including systems designed for multimodal inputs, long videos, or efficient long-context processing.Source: D BASELINES DETAILS · p. 18
• The RL-based comparison includes Video-R1, while FrameMind learns a dynamic perception policy from Qwen2.5-VL-7B.Source: D BASELINES DETAILS · p. 18
• Proprietary comparison models include GPT-4V, GPT-4o, and Gemini 1.5 Pro.Source: D BASELINES DETAILS · p. 18

E.1 PROMPTING FORMAT

• Each FiCOT turn receives the question and interaction history, including prior reasoning, tool calls, and timestamped visual evidence.Source: §E.1 · p. 18
• The prompting protocol alternates tool use and reasoning with turn summaries, then uses gathered visual evidence to answer or request further video information.Source: §E.1 · pp. 19–20

E.3 REWARD MANAGER

• The reward manager scores accuracy using exact match for questions with a definitive answer and an LLM judge for open-ended descriptive answers.Source: §E.3 · p. 20
• GPT-4o mini judges open-ended answers by accepting equivalent phrasing while assessing factual accuracy and completeness against the ground truth.Source: §E.3 · p. 21

E.4 ANALYSIS OF TRAINING PARADIGMS AND DATA EFFICIENCY

• FrameMind uses pure reinforcement learning with outcome-based rewards, unlike SFT baselines and Video-R1’s SFT-then-RL approach.The model learns a reasoning policy without explicit instruction-response pairs.Source: §E.4 · p. 21
• FrameMind trains exclusively on video-text data, while most baselines combine image-text and video-text data.Source: §E.4 · p. 21