PaperlayerView exact revision on arXiv

Source-linked AI summary

Video-HopChain: Multi-Hop Questions and Confidence-Gated Exploration for Video Reasoning Models

Trung Nguyen Quang, Yuhao Dong, Shuo Sun, Shuai Liu, Shulin Tian, Kim-Hui Yap, Ziwei Liu

arXiv:2609.25773v1cs.CVcs.AI

TL;DR

Multi-hop data synthesis has been explored for still-image reasoning, but not yet for video. Video-HopChain introduces multi-hop video questions, and Confidence-Gated Exploration (CGE) changes how zero-variance GRPO groups are sampled. Video-HopChain training raises the mean across eight benchmarks from 55.4 to 57.9, with every benchmark improving; CGE raises it further to 59.3.

  • Problem

    Although RLVR video question answering is growing, the multi-hop data-synthesis framework used for still images has not yet been explored for video.

    Source: §1 · p. 2

  • Method

    Video-HopChain builds verifiable video questions chaining three to six moment-based yes/no hops, while CGE confidence-gates masked exploration of zero-variance GRPO groups.

    Source: §1 · p. 2

  • Results

    55.4 to 57.9 mean across eight benchmarks after Video-HopChain training, with every benchmark improving; CGE raises the mean further to 59.3.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Video-HopChain training and CGE offer complementary improvements to GRPO video reasoning, with CGE recovering zero-variance groups without additional rollouts.

    Source: §1 · p. 2, §6 · p. 9

    The study trains one base model at one scale, so whether its results hold at other scales remains untested.

    Source: B LIMITATIONS · p. 19

Abstract

from arXiv · show

HopChain has shown on still images that multi-hop data synthesis improves vision-language reasoning, because long chain-of-thought reasoning exposes errors that compound across steps, while most data used for reinforcement learning with verifiable rewards (RLVR) rarely demands a chain of visual evidence, so these weaknesses are likely to stay unexposed. We observe the same problem in video, where this framework has not yet been explored. We therefore build Video-HopChain, a dataset of 22,550 multi-hop video questions over 13,378 videos, together with a held-out benchmark of 1,000 questions. Each question chains three to six yes/no questions about moments in one video, and each yields one of two integers depending on its answer. The final answer is the sum of these integers, so an exact match on that sum gives the verifiable reward that RLVR needs. We first train Qwen3-VL-8B with GRPO on a standard video dataset, and a second stage on Video-HopChain then raises the mean over eight video understanding and reasoning benchmarks from 55.4 to 57.9 and improves every one of them. Training on such a dataset, however, exposes a known limitation of GRPO: its learning signal comes from the reward variance within a group, so hard questions whose rollouts are all incorrect and easy questions whose rollouts are all correct both leave the group with no gradient. To recover these groups at the same compute budget, we introduce Confidence-Gated Exploration (CGE). With 8 rollouts per question, CGE samples the first 4 as usual. If these 4 are either all correct or all incorrect, it samples the last 4 with the policy's most confident token masked inside the reasoning span, and removes the masked positions from the loss while all 8 rollouts enter the advantage. With CGE, the mean rises further to 59.3. We release the dataset, the checkpoint, and the data generation and training code.

1 INTRODUCTION

Video-HopChain makes video reasoning verifiable through multi-hop questions that require evidence across moments, and CGE targets GRPO groups that otherwise provide no learning signal. Training with the dataset and CGE raises benchmark means to 59.3.

1 INTRODUCTION

  • Video RLVR often has verifiable answers but lacks questions requiring a chain of visual evidence, leaving models’ visual reasoning weaknesses underexposed.Prior audits report that models may answer without looking at the video or rely on priors despite encoding visual input.Source: §1 · p. 2
  • Video-HopChain provides 22,550 training questions over 13,378 videos and a benchmark of 1,000 questions on 1,000 held-out videos.Questions chain three to six yes/no hops about video moments, assign an integer for each answer, and sum those integers for an exact-match-verifiable result; generation uses shot captions to produce questions.Source: §1 · pp. 2–3
  • 59.3 mean across eight video benchmarks with CGE, following 57.9 with Video-HopChain and 55.4 before that; Video-HopChain improved every benchmark.The reported progression is 55.4 to 57.9 after further training on Video-HopChain, then 57.9 to 59.3 with CGE.Source: §1 · p. 2
  • CGE targets zero-variance GRPO groups by conditionally masking a confident reasoning token in half of eight rollouts, excluding masked positions from loss while using all eight for advantage.After four normal rollouts are all correct or all incorrect, the remaining four use a top-token mask when one token has more than τ = 0.95 probability; no extra rollouts are added.Source: §1 · p. 2

2 VIDEO-HOPCHAIN

Video-HopChain builds multi-hop video questions whose answers depend on linked evidence across moments and whose yes/no responses determine a uniquely encoded sum. Its generation pipeline verifies candidate questions and filters out those the base model already solves easily, producing a video-disjoint held-out split.

2.1 QUESTION DESIGN

  • Questions chain 3–6 yes/no hops over one video, using order, spatial, action, or attribute evidence, with selector chains chosen 70% of the time.In selector questions, an earlier answer determines which moment a later hop asks about; random integers from 1 to 80 encode hop answers in a plain sum, with unique totals for answer combinations.Source: §2.1 · pp. 3–4

2.2 GENERATION PIPELINE

  • The pipeline keeps videos at least three minutes long, captions every shot, and uses Qwen3.8-27B to generate two candidate questions per video.The hop count, hop types, links, and number assignments are specified before question generation.Source: §2.2 · pp. 4–5
  • Candidates are checked against captions, rewritten up to three times if faulty, then dropped if still faulty or if the base model solves them in 6 of 8 rollouts.This verification and difficulty filtering retains questions that remain challenging for the policy.Source: §2.2 · p. 5

2.3 STATISTICS AND SPLIT

  • The dataset contains 22,550 training questions over 13,378 videos and 1,000 held-out questions from 1,000 further videos, split by video.The held-out set is drawn from longer, more varied questions and stratified by hop count, video length, and source.Source: §2.3 · p. 5

3 CONFIDENCE-GATED EXPLORATION

CGE addresses GRPO groups with no reward variance by changing how the second half of rollouts is sampled. It masks highly confident reasoning tokens while keeping those positions out of the loss and computing advantages across the full group.

3.1 PRELIMINARIES

  • GRPO standardizes each rollout’s reward within its group to form its advantage, so a group with identical rewards has zero advantages and contributes no gradient.The reward combines answer correctness and response-format compliance.Source: §3.1 · p. 5

3.2 CONFIDENCE-GATED EXPLORATION

  • CGE resamples the second four of eight rollouts when the first four are all correct or all incorrect, intervening on both zero-variance cases.The gate uses answer accuracy rather than the combined reward, so format alone cannot make a group appear to have variance; intervening only on all-incorrect groups performs worse.Source: §3.2 · pp. 5–6
  • In the second wave, CGE masks the top-probability token when its probability exceeds 0.95 inside the reasoning span, sampling from the renormalized alternatives.The mask leaves answer tokens and the think tags unchanged; elsewhere, sampling uses the original policy.Source: §3.2 · pp. 6–7
  • CGE excludes masked positions from the loss but computes advantages over all eight rollouts; if the second wave changes the outcome, every rollout receives a gradient.If the group remains zero-variance, it costs no more than before.Source: §3.2 · pp. 6–7
  • CGE adds one barrier per prompt, but the asynchronous trainer shows no significant throughput change.The second wave must wait for the first wave to be scored.Source: §3.2 · p. 7

4 EXPERIMENTS

Video-HopChain improves results across eight public video benchmarks, and CGE raises the mean further. V-HopChain also leads on most benchmarks against ten open-source video reasoning models.

4.1 SETUP

  • Training uses 105,993 general-dataset rows or 22,550 Video-HopChain rows, with Video-HopChain training staged after a general-data checkpoint for the second-stage runs.The general mixture combines five public datasets; second-stage Video-HopChain runs use either plain GRPO or CGE from the same first-stage checkpoint.Source: §4.1 · pp. 7–8
  • Evaluation covers eight public video benchmarks and 1,000 held-out Video-HopChain questions, using consistent benchmark settings and 100 evenly sampled frames per video.Training samples 140 frames for Video-HopChain and 24 for the general dataset; CGE uses τ = 0.95 and a first wave of four rollouts.Source: §4.1 · pp. 7–8

4.2 MAIN RESULTS

  • On eight public benchmarks, Video-HopChain raises the mean and improves all eight; adding CGE raises it again and gives the combined run every best column, tying Video-HopChain on Video-Holmes.The main-results comparison is summarized in Table 1.Source: §4.2 · p. 8, Table 1 · p. 8
  • V-HopChain leads on most benchmarks against ten open-source video reasoning models, ties Conan on VRBench, and places second on Video-Holmes and LongVideo-Reason behind OneThinker.The comparison covers released systems rather than recipes because their base models, training data, and compute differ.Source: §4.2 · p. 8, Table 2 · p. 8

5 ANALYSIS

CGE restores reward variance for more groups at the same compute budget and keeps policy entropy above plain GRPO. Its effects vary with question type, and it slightly lengthens the trained policy’s responses.

5 ANALYSIS

  • CGE yields about half again as many gradient-bearing groups at the same compute budget by restoring variance in one zero-variance group in six.The restored-variance share grows from about one group in seven to one in five; the second wave solves questions the first wave never solves and answers as accurately as the first wave.Source: §5 · p. 9
  • CGE keeps policy token entropy a few tenths of a nat above plain GRPO throughout training, while both runs reach the same training reward.Neither run reaches entropy collapse, but CGE retains more room to explore; the authors attribute the margin to masked positions and the tokens that follow them.Source: §5 · p. 9
  • CGE responses average 810 tokens against 708 for plain GRPO, while its masked second wave averages 696 tokens against 923 for the first wave.Both runs settle between 600 and 900 tokens once the format is stable and lengthen again late in training; the authors attribute CGE’s extra length to the policy it trains and hypothesize that masking moves it off its longest paths.Source: §5 · p. 9
  • On Video-HopChain, the second wave restores variance in more intervened groups than on the general dataset, which the authors attribute to multi-hop chains’ many decision points.Because masking acts only inside the reasoning span, longer spans offer more positions to change; a multiple-choice question about a single shot offers fewer decision points.Source: §5 · p. 9

6 CONCLUSION

Video-HopChain improves video understanding and reasoning over training on general video reasoning datasets, while Confidence-Gated Exploration further improves results by resampling zero-variance groups without additional rollouts.

6 CONCLUSION

  • Video-HopChain training improves general video understanding and reasoning more than training on general video reasoning datasets.The dataset consists of multi-hop questions over videos.Source: §6 · p. 9
  • Confidence-Gated Exploration resamples the second half of zero-variance groups under a top-token mask, recovering those groups without additional rollouts.Source: §6 · p. 9
  • Confidence-Gated Exploration further improves results after Video-HopChain training.Source: §6 · p. 9

AI USE STATEMENT … C CONFIGURATION

The paper discloses its use of generative AI, situates Video-HopChain and CGE among related work, and documents the study’s limitations and appendices. It notes that only one model scale was tested and that the default and CGE video runs differ only in exploration settings.

AI USE STATEMENT

  • The authors used generative AI to polish and grammar-check the paper, help write code, and build the synthetic dataset; they checked the results and take responsibility.For Video-HopChain, a vision-language model captions each video shot and a language model writes and verifies each question.Source: AI USE STATEMENT · p. 10

APPENDIX CONTENTS

  • Appendices A–E cover related work, limitations, configuration, image-data ablations, and image benchmark results.Appendix D examines threshold τ and intervention groups on image data; Appendix E reports effects on general image ability.Source: APPENDIX CONTENTS · pp. 17–18
  • Appendices F–K cover dataset details, second-wave group and token analyses, derivations, update effects, and computational cost and response length.Appendix F includes statistics, row format, system prompt, and held-out split; Appendix I derives entropy bounds and the importance ratio at a masked position.Source: APPENDIX CONTENTS · p. 18

A RELATED WORK

B LIMITATIONS

  • The study tests one base-model scale, relies on automatically generated and verified data, and omits ablations of G1, reasoning-span masking, and adaptive τ.The authors note that results may differ at other scales or with manually labelled and verified data.Source: B LIMITATIONS · p. 19

D ABLATIONS

The ablations test CGE’s confidence threshold and which zero-variance groups to intervene on, using controlled image-dataset runs rather than measuring video accuracy.

Ablation setup

  • The ablations isolate CGE on 74K-sample OpenMMReasoner image data, holding the model, algorithm, reward, and compute fixed; scores are validation means over six benchmarks.Image runs provide more validation points per wall-clock budget; the results are evidence about CGE itself, not measurements of video accuracy.Source: D ABLATIONS · p. 21

Threshold

  • More than five points above plain GRPO, τ = 0.90 and 0.95 stay within 0.1 points of each other at matched compute.Both thresholds learn the answer format; Figure 6 reports the best mean accuracy across six OpenMMReasoner image benchmarks.Source: D ABLATIONS · p. 21, Figure 6 · p. 22
  • At τ = 0.85 and 0.80, masking ordinary-uncertainty positions prevents reasoning from closing, inflates responses beyond 13,000 tokens, clips 0.69 of tokens, and degrades answer-format completion.The mask acts only inside the reasoning span, so the format degrades because reasoning does not end, not because answer tokens are deleted.Source: D ABLATIONS · p. 21

Choice of the intervened groups

  • Intervening only on all-incorrect groups reduces intervention from 0.79 to 0.29; the run peaks at 66.7, 5.8 points below CGE and no better than plain GRPO, before collapsing.Most zero-variance groups are all-correct, so CGE intervenes on both all-correct and all-incorrect groups.Source: D ABLATIONS · p. 21, Table 4 · p. 21

E IMAGE BENCHMARK RESULTS

The section evaluates whether video-only training lowers general image ability by testing the same checkpoints on six public image benchmarks. Scores use exact match, and Table 5 reports image-understanding accuracy.

E IMAGE BENCHMARK RESULTS

  • The same checkpoints are evaluated on six public image benchmarks, covering 11,527 questions in total.The benchmarks are AI2D, MathVista, MMBench-EN, MMMU, MMStar, and RealWorldQA.Source: E IMAGE BENCHMARK RESULTS · p. 22
  • All benchmarks are scored by exact match on a multiple-choice letter or numeric answer, without using a model as judge.The evaluation uses AI2D's multiple-choice split, MathVista's testmini split, MMBench's English dev split, and MMMU's multiple-choice validation questions.Source: E IMAGE BENCHMARK RESULTS · p. 22
  • Table 5 reports image-understanding accuracy in percent and gives the question count for each benchmark.The caption indicates that each column's best value is bolded and its second-best value is underlined; the supplied passage does not include the table values.Source: Table 5 · p. 22

F DATASET DETAILS

This appendix documents Video-HopChain’s training-row structure, shared evaluation prompt, and video-based held-out split. The held-out questions are selected and stratified by hop count, video length, and source.

Row format

  • Each training row contains a system-and-user prompt, video path and decode contract, string answer, and metadata; hop records and caption quotes are stored separately for auditing.The metadata records the answer, hop count and types, and question and video identifiers; hop records include moment references, mappings, values, and caption quotes.Source: F DATASET DETAILS · p. 23

System prompt

  • Training and held-out rows use the same system prompt, requiring reasoning in <think> tags and a boxed final answer in <answer> tags, with no text outside them.For multiple-choice answers, the prompt specifies using the letter inside the boxed answer.Source: F DATASET DETAILS · p. 23

Held-out split

  • The held-out split selects one question per video with at least four hops and two hop types, then stratifies by hop count, video length, and source.Its 1,000 questions include 302 four-hop, 450 five-hop, and 248 six-hop questions; video-length counts are 151, 301, 300, and 248 across the listed bins, while source counts are 467 LongVILA, 358 FineVideo, and 175 Vript.Source: F DATASET DETAILS · p. 23

G GROUP-LEVEL ANALYSIS OF THE SECOND WAVE

The second wave of CGE adds reward variance mainly to groups it intervenes on, while achieving accuracy comparable to the first wave.

The variance the second wave adds

  • The second wave adds more reward variance to intervened groups than to groups the mask leaves alone.The added variance rises from 0.020 early to 0.028 later on intervened groups, versus 0.005 to 0.009 on untouched groups; early and later denote the first and second halves of the measured window.Source: G GROUP-LEVEL ANALYSIS OF THE SECOND… · pp. 23–24, Figure 7 · p. 24

The accuracy of each wave

  • The second wave is about as accurate as the first overall and more accurate on intervened groups.Over all groups, first- and second-wave accuracies are 0.17 and 0.16; on intervened groups, the second wave is higher by 0.02 early and 0.01 later. On the general dataset, the first and second waves reach 0.81 and 0.80, respectively; the second wave follows a path the policy otherwise would not have taken.Source: G GROUP-LEVEL ANALYSIS OF THE SECOND… · pp. 24–25, Figure 7 · p. 24, Table 7 · p. 24

H TOKEN-LEVEL ANALYSIS OF THE MASK · I DERIVATIONS · J EFFECT OF THE SECOND WAVE ON THE UPDATE

The analyses characterize what CGE masks and substitutes, derive how masking changes token entropy and update ratios, and show how the second wave affects logged updates while training reward and gradient scale remain similar.

H TOKEN-LEVEL ANALYSIS OF THE MASK

  • The mask removes mostly answer letters, “boxed” and “answer,” and function words; replacements are more often emphasis markers or punctuation than alternate object names.The mask examines positions with top-1 probability above τ at 2.2% of positions, and it targets answer drafts inside the reasoning span, not the rewarded answer after the closing think tag.Source: H TOKEN-LEVEL ANALYSIS OF THE MASK · p. 25
  • 0.13 of all-incorrect groups gained a correct rollout, and 0.12 gained a solution the first wave had not found.Substitutions changed hop content such as answer letters, frame sides, directions, colors, and person parts, as well as relations, claims, or chain-step labels.Source: H TOKEN-LEVEL ANALYSIS OF THE MASK · p. 25, Table 8 · p. 26
  • The median masked position was token 242, about one in three masked positions fell beyond token 400, and the policy’s second-ranked token was sampled at half of positions.The sampler drew one of the first three candidates at about two thirds of positions; at 46% of positions, the second-most-likely token held more than half the probability left after masking.Source: H TOKEN-LEVEL ANALYSIS OF THE MASK · pp. 25–26, Table 9 · p. 26

I DERIVATIONS

  • At p*=0.80, entropy spans 0.50–2.89 nats, so top-1 probability alone determines entropy only within a wide range.The intervals are [0.33, 1.52] at p*=0.90, [0.20, 0.80] at p*=0.95, and [0.06, 0.18] at p*=0.99; conversely, at H=0.5 nats, p* can lie in [0.80, 0.97].Source: I DERIVATIONS · pp. 26–27
  • After masking the top token, the renormalized distribution is q(y)=π(y)/(1-p*) for y≠y*; at the first update, a masked sampled token’s importance ratio is 1-p*.The trace stores p* to five decimals, so the post-mask entropy form is evaluated only when 1-p*>10^-3; later updates move the ratio as for other tokens, while unmasked positions sampled from πθ have ratio one.Source: I DERIVATIONS · p. 27

J EFFECT OF THE SECOND WAVE ON THE UPDATE

  • CGE’s clip-range removals were 0.023 and 0.021 versus plain GRPO’s 0.005 and 0.009, while second-half training reward was 0.360 versus 0.355.The advantage magnitude was larger under CGE early in training, while the gradient norm stayed within 0.14 of plain GRPO over the full window; the clip absorbed perturbations rather than removing their rollouts.Source: Table 10 · p. 27, J EFFECT OF THE SECOND WAVE ON THE… · p. 27

K COMPUTATIONAL COST AND RESPONSE LENGTH

CGE adds little measured training cost at the fixed budget of eight rollouts per question, while its trained policy produces longer responses than plain GRPO. The second wave is shorter than the first, and neither run incurs measurable reward loss from response truncation.

K COMPUTATIONAL COST AND RESPONSE LENGTH

  • 0.15 to 0.58 seconds: CGE’s advantage-and-check step grew, while other timing rows differed by at most 2.0% and a training step exceeded 2,000 seconds.Token throughput differed by 1.4%, and the runs had similar model utilization and rollout-wait time. CGE keeps eight rollouts per question and applies the mask as a logits processor, adding no forward passes; asynchronous sampling hides the wait for first-wave rewards.Source: K COMPUTATIONAL COST AND RESPONSE LENGTH · p. 28, Table 11 · p. 28
  • 810 tokens against 708: CGE responses were longer on average than plain GRPO, but its second-wave responses were shorter than its first-wave responses, 696 against 923 tokens.The first CGE wave, which carries no mask, was already longer than plain GRPO; the trained CGE policy retained longer responses at evaluation, with the mask disabled for both checkpoints.Source: K COMPUTATIONAL COST AND RESPONSE LENGTH · p. 28
  • Fewer than 0.4% of responses in either run reached the 16,384-token limit, so truncation imposed no measurable reward cost.Source: K COMPUTATIONAL COST AND RESPONSE LENGTH · p. 28

Related papers

Loading 2609.25773v1…