ReWatch-R1: Boosting Complex Video Reasoning in Large Vision-Language Models through Agentic Data Synthesis

• ReWatch combines temporally dense captions, high-difficulty question-answer pairs, and video-grounded reasoning traces generated through a multi-agent re-watching process.The traces explicitly document information retrieval and verification against video content.Source: §1 · p. 2
• The model is post-trained with SFT and RLVR, using O&R rewards to assess final-answer correctness and whether intermediate observations are video-supported and sufficient.The reward explicitly incentivizes evidence-linked reasoning and is described as reducing hallucinations and improving logical consistency.Source: §1 · pp. 2–3
• ReWatch-R1 sets state-of-the-art on five challenging video reasoning benchmarks, substantially outperforming models trained on alternative open-source data.Source: §1 · p. 3

2 DATA CONSTRUCTION: THE REWATCH DATASET

• ReWatch contains 10k captions, 170k QA pairs, and 135k CoTs produced through hierarchical captioning, high-difficulty QA generation, and multi-agent CoT synthesis.Source: §2 · p. 3

2.1 STAGE 1: HIERARCHICAL VIDEO CAPTIONING

• Hierarchical captioning segments videos semantically, generates high-frame-rate event descriptions with relative timestamps, realigns them to absolute time, and combines them into precise captions while avoiding long-video hallucinations.Source: §2.1 · p. 3

2.2 STAGE 2: HIGH-DIFFICULTY QA PAIR GENERATION

• Contrastive prompts require questions answerable from detailed captions but not summaries; three filters verify answers and remove general-knowledge and summary shortcuts before multiple-choice expansion.Source: §2.2 · p. 4

2.3 STAGE 3: MULTI-AGENT CHAIN-OF-THOUGHT SYNTHESIS

• A Reasoner chooses thoughts and actions from the interaction history, while an Observer retrieves caption information; their trajectory is converted into tagged natural-language CoT for training and reward calculation.The loop continues until the Reasoner produces a final answer.Source: §2.3 · pp. 4–5

3 POST-TRAING ON REWATCH DATASET

• Qwen2.5-VL is trained with SFT followed by RL to produce ReWatch-R1.The RL stage uses GRPO and the proposed O&R reward.Source: §3 · p. 5, §3.2 · p. 6

3.1 SUPERVISED FINE-TUNING STAGE

• SFT jointly trains video-text alignment, direct answering, and step-by-step reasoning on three ReWatch datasets using distinct response-mode prompts.Concurrent optimization produces a model proficient in direct answering and complex reasoning for the subsequent RL stage.Source: §3.1 · pp. 5–6

3.2 REINFORCEMENT LEARNING STAGE

• Because final-answer accuracy alone overlooks video-grounded reasoning, O&R rewards factual observations and reasoning sufficient to recover the answer.The paper identifies potential visual or linguistic hallucinations as a concern with accuracy-only rewards.Source: §3.2 · p. 6
• The observation reward compares CoT observations with detailed captions, while the reasoning reward judges answers inferred from the model's actions and observations.The model output is parsed to extract actions and observations; an LLM answers from them, and a judge assesses answer consistency.Source: §3.2 · p. 6
• A format reward encourages tagged actions, observations, and answers, and GRPO is used for model optimization.The expected tags are <action>, <observation>, and <answer>.Source: §3.2 · p. 6
• Table 1 compares performance on video-reasoning tasks and marks the best results among models of the same size.Its caption notes reproduced models use 192 frames and † marks RL conducted with the same data as ReWatch-R1.Source: Table 1 · p. 7

4.1 MAIN RESULTS

• In both 192-frame and 384-frame settings, ReWatch-R1’s average scores across five reasoning benchmarks surpass those of all comparable-size models.The experiments compare ReWatch-R1 with other LVLMs on five video-reasoning and four video-understanding benchmarks.Source: §4.1 · p. 7, §4 · p. 7
• ReWatch-R1-SFT scores 33.25%, above Video-R1-SFT (29.74%) and LongVideoReason-SFT (26.31%); RL raises ReWatch-R1 to 35.51%.The authors attribute the SFT-only comparison to the quality of ReWatch-CoT data, and describe RL as improving on the SFT version.Source: §4.1 · p. 8
• For the untrained base model, thinking yields 27.54% versus 30.71% for direct answering; the fully trained ReWatch-R1 excels with thinking.The paper reports that enabling CoT can induce hallucinations in the untrained model.Source: §4.1 · p. 8

4.2 ANALYSIS RESULTS

• Figure 5 ablations show that training without SFT causes a catastrophic performance drop, while replacing ReWatch-CoT with Video-R1 CoT significantly degrades performance.The authors describe SFT as a prerequisite for RL and ReWatch-CoT as higher-quality training data for complex reasoning.Source: Figure 5 · p. 8, §4.2 · p. 8
• 42.0% all, 34.3% reasoning, and 51.7% understanding were the lowest scores with baseline QA; ReWatch-QA provides notable improvements.The baseline uses Video-R1-QA (10k) and LongVideoReason-QA (10k).Source: §4.2 · p. 8, Figure 5 · p. 8
• 3.31 vs. 1.82 reasoning steps and 398.75 vs. 205.74 response length distinguish ReWatch-QA from Video-R1-QA; text-only accuracy is 29.4% vs. 68.9%.ReWatch's 29.4% text-only accuracy is near the 25% random-guess baseline.Source: Figure 6 · p. 8, §4.2 · p. 9
• Figure 6b shows accuracy rising while average action counts fall during RL, after SFT increases both as it teaches structured reasoning.The authors interpret the RL trend as refining the policy by pruning redundant steps.Source: Figure 6 · p. 8, §4.2 · p. 9
• Figure 7 shows thinking develops more slowly during SFT, then makes a performance leap during RL and ultimately surpasses non-thinking on every task.The authors describe the SFT phase as teaching reasoning format and RL as helping the model learn causal links between reasoning and correct answers.Source: Figure 7 · p. 9, §4.2 · p. 9

5 CONCLUSION

• ReWatch-R1 establishes state-of-the-art results on five challenging video reasoning benchmarks.Source: §5 · p. 9
• ReWatch combines temporally dense captions, challenging multi-hop questions, and video-grounded Chain-of-Thought traces in a multi-stage agentic pipeline.Source: §5 · p. 9
• The model is post-trained with SFT and RLVR, using an Observation & Reasoning reward that assesses final-answer correctness and reasoning grounding.Source: §5 · p. 9

A.1 DATASET STATISTIC

• The authors manually defined 10 question types for the dataset.Source: §A.1 · p. 14
• Table 2 presents ReWatch dataset statistics, while Figure 8 provides distribution information.Source: Table 2 · p. 14, §A.1 · p. 14

A.2 MODEL SETTINGS FOR DATA SYNTHESIS

• Figure 8 presents the distribution of the dataset.Source: Figure 8 · p. 14
• ReWatch-Caption uses Gemini2.5-Flash Non-Thinking for semantic segmentation and detailed description generation.Source: §A.2 · p. 14
• ReWatch-QA uses Gemini2.5-Flash-Lite Non-Thinking for summaries, Gemini2.5-Flash Thinking for contrastive QA, and GPT4.1 for answer verification.Source: §A.2 · p. 14
• ReWatch-CoT uses Gemini2.5-Flash Thinking as reasoner, GPT4.1 as observer, and Gemini2.5-Flash-Lite Non-Thinking to convert trajectories.Source: §A.2 · p. 14

B.1 EXPERIMENTAL SETUP

• Evaluation covers five video reasoning benchmarks and four video general-understanding benchmarks, using accuracy assessed against ground truth.Source: §B.1 · p. 15
• ReWatch-R1 starts from Qwen2.5-VL-7B-Instruct, receives SFT on three ReWatch datasets, then RL on 40k QA pairs sampled from three sources.Source: §B.1 · p. 15
• Both training stages use a 16k context, 2.0 fps, and at most 192 frames; SFT runs 10 epochs and RL runs one.Source: §B.1 · p. 15
• Comparisons include six video reasoning models and SFT/RL reproductions using Video-R1-CoT and LongVideoReason-CoT data.Source: §B.1 · p. 15
• Inference uses greedy decoding for the listed Qwen, Video-R1-family, and LongVideoReason/ReWatch models, with different temperatures for GLM4.1V-9B and InternVL3.5-8B.Source: §B.1 · p. 15

B.2 PERFORMANCE COMPARISON ON VIDEO UNDERSTANDING BENCHMARKS

• 54.15% average at 192 frames across four general video-understanding benchmarks places ReWatch-R1 at state-of-the-art among comparable-size models.Table 3 compares performance on video-understanding tasks.Source: §B.2 · p. 15, Table 3 · p. 16
• After SFT, ReWatch-R1-SFT still lagged the base model’s direct-answer performance, but RL gains let ReWatch-R1 surpass that baseline on video-understanding tasks.The authors report that this pattern suggests reasoning improvements generalize to foundational understanding tasks.Source: §B.2 · p. 16

B.3 PERFORMANCE COMPARISON ACROSS DIFFERENT VIDEO DURATIONS

• 27.46% on reasoning tasks for videos over 20 minutes exceeds LongVideoReason-RL’s 24.03% by over 3.4 percentage points.Figure 9 compares performance across short (0–3 minutes), medium (3–20 minutes), and long (over 20 minutes) videos; results use 192 frames.Source: Figure 9 · p. 17, §B.3 · p. 17
• Reasoning performance declines as video duration increases, but ReWatch-R1’s reported score drops from 40.38% on short videos to 27.46% on long videos.The authors describe its rate of degradation as more attenuated than the general trend.Source: Figure 9 · p. 17, §B.3 · p. 17

B.4 COMPARATIVE ANALYSIS OF DATASET-INDUCED REASONING COMPLEXITY AND VIDEO DEPENDENCY

• 3.31 vs. 1.82 reasoning steps and 398.75 vs. 205.74 characters: ReWatch samples elicited longer, more multi-step outputs than Video-R1 samples.The analysis used 5,000 correctly answered samples randomly selected from each dataset’s outputs.Source: §B.4 · p. 17
• Text-Only Accuracy was 29.4% for ReWatch and 68.9% for Video-R1, with ReWatch’s result close to the 25% random-guessing baseline.Text-Only Accuracy measures Qwen2.5-VL-7B’s accuracy when answering with text alone and no video.Source: §B.4 · p. 17
• The analysis measured average reasoning steps, average response length, and Text-Only Accuracy on correctly answered samples from the two datasets.It used ReWatch-R1-SFT outputs on ReWatch training data and the multiple-choice subset of Video-R1 training data.Source: §B.4 · p. 17

C.1 VIDEO QA DATASETS AND BENCHMARKS

• Benchmarks target causal attribution, temporal ordering, state tracking, counting, and cross-modal grounding, while long-video suites emphasize hour-scale contexts and dense event structure.The cited evaluations report that multi-hop, evidence-driven video reasoning remains underdeveloped.Source: §C.1 · p. 18
• Training corpora often lack precise temporal annotations, multi-step questions, and verifiable video-grounded CoT; ReWatch pairs hierarchical captions, shortcut-filtered QA, and retrieval-verification traces.The authors describe this combination as designed to provide process-level supervision and question difficulty for complex video reasoning with RLVR.Source: §C.1 · p. 18

C.2 VIDEO REASONING IN LARGE VISION-LANGUAGE MODELS

• Prior video RL typically emphasizes final-answer rewards, while existing agentic synthesis can lack iterative think-and-act; ReWatch couples agentic data synthesis with RLVR and internalizes grounded reasoning.The cited work also describes training-free agentic methods and context-intensive “think with video” approaches as having distinct limitations.Source: §C.2 · p. 18

D.1 THINKING PROMPTS

• Evaluation used different thinking prompts by model, including Prompt 11 for ReWatch-R1 and Prompt 10 for Qwen2.5-VL.The settings also specify prompts for GLM4.1V, InternVL3.5, Video-R1, VideoRFT, Video-Chat-R1, and LongVideoReason.Source: §D.1 · p. 19

D.2 NON-THINKING PROMPTS

• All models used Prompt 16 for non-thinking evaluation; ReWatch-R1-SFT training used Prompts 17, 16, and 11 for Caption, QA, and CoT, respectively.Source: §D.2 · p. 21

D.3 ANSWER JUDGE PROMPT

• The answer judge checks whether the model answer is consistent with the standard answer, counting equivalent meanings as consistent.Source: §D.3 · p. 21