PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Agent Self-Improving Reinforcement Learning for Video Reasoning

Mingwen Zhang, Jisheng Dang, Minqiang Yang, Bimei Wang, Bin Hu, Tat-Seng Chua

arXiv:2608.28675v1cs.CVcs.CL

TL;DR

Video reasoning needs evidence segments that support queries, but training often relies on local temporal objectives while verification is reserved for inference. MASIRL trains a Grounder with a frozen Verifier through relative policy optimization and calibration, achieving consistent zero-shot transfer gains across grounded QA, temporal grounding, and long-video QA. The evidence supports frozen verification as a training signal, although strict boundary precision remains weaker.

  • Problem

    Video reasoning requires selecting query-supporting temporal evidence, while existing training often uses local boundary or span objectives and inference-only verification.

  • Method

    MASIRL couples a trainable Grounder with a frozen Verifier whose segment scores guide group-relative policy learning, pseudo-target calibration, and update gating.

  • Results

    A 2B model transfers zero-shot across grounded QA, temporal grounding, and long-video QA, with modest but consistent gains over the strongest same-scale baseline, especially on relevance-oriented metrics.

  • Takeaways & Limitations

    Within the tested zero-shot setting, frozen verification can serve as a training signal for video evidence selection.

  • Takeaways & Limitations

    Strict boundary precision remains weaker because the frozen Verifier emphasizes segment relevance rather than exact boundary discrimination, and same-scale gains are small.

Abstract

from arXiv · show

Video reasoning tasks such as grounded video question answering and temporal grounding require selecting temporal evidence that supports the query. In many current training setups, temporal supervision is applied through local objectives such as boundary regression or span generation, while verification is used mainly to rerank candidate segments at inference time. We study whether a frozen verifier can also guide training. Our multi-agent framework couples a trainable \emph{Grounder} with a frozen \emph{Verifier}: the Grounder samples candidate trajectories and evidence segments, the Verifier assigns query-conditioned segment scores, a group-relative policy-gradient objective favors trajectories that outperform their within-input peers, and a bootstrapped calibration loss steers temporal predictions toward verifier-preferred spans. Trained on source tasks and evaluated without target-dataset fine-tuning, a two-billion-parameter instantiation transfers zero-shot across grounded question answering, temporal grounding, and long-video question answering, reaching 28.7\% intersection-over-union and 25.4\% answer-grounding accuracy on a grounded-question-answering benchmark, 46.1\% intersection-over-union on a temporal-grounding benchmark, and 54.1\% on a long-video question-answering benchmark. Relative to a strong same-scale baseline, the gains are modest but consistent, with the clearest improvements on relevance-oriented metrics such as intersection-over-union and moderate-overlap recall. Within the tested benchmarks and transfer setting, the results support frozen verification as a training signal for evidence selection, while showing that strict boundary precision remains comparatively weaker. Code and models are available at https://anonymous.4open.science/r/MASIRL-E50C/

I. INTRODUCTION

The paper frames video reasoning as selecting temporal evidence that genuinely supports a query, identifying a gap in local boundary supervision and inference-only verification. MASIRL moves frozen verification into training, combining relative policy learning and temporal calibration, and transfers zero-shot across three tasks with modest, relevance-oriented gains.

  • Motivation: Video reasoning requires identifying the temporal interval that supports a query while excluding semantically related distractors.The paper treats grounded interpretation and temporal localization as an evidence-selection problem.
  • Motivation: Existing pipelines use local boundary or span objectives, while verifiers often influence only inference-time candidate selection.This leaves limited direct preference among semantically plausible intervals and weak sequence-level credit assignment.
  • Results: A 2B model transfers zero-shot across grounded QA, temporal grounding, and long-video QA without target-dataset fine-tuning.It reaches 28.7 mIoU and 25.4% Acc@GQA on NExT-GQA, 46.1 mIoU on Charades-STA, and 54.1% on Video-MME.
  • Method: MASIRL trains a Grounder with scores from a frozen, query-conditioned Verifier rather than restricting verification to reranking.The Verifier evaluates explicit candidate segments while remaining decoupled from Grounder updates.
  • Method: The framework combines group-relative REINFORCE, bootstrapped boundary calibration, and verifier-based update gating.These components jointly support sequence-level policy updates and temporal refinement.

II. RELATED WORK

Related work covers temporal moment retrieval, generative video-language grounding, reinforcement learning, and learned verification. MASIRL differs by using a frozen verifier during training to score candidate evidence segments and guide the Grounder’s policy and temporal predictions.

  • Temporal Sentence Grounding: Temporal sentence grounding localizes the interval in an untrimmed video corresponding to a natural-language query.Prior approaches include proposal-based, matching-based, and span-based paradigms.
  • Generative Video-Language Models: Generative video-language models jointly produce answers and temporal spans, but their supervision remains largely based on span prediction or supervised generation.MASIRL instead introduces a frozen verifier as a training-time signal for choosing among candidate evidence segments.
  • Reinforcement Learning: Reinforcement learning provides a mechanism for optimizing non-differentiable localization criteria such as IoU through sequential decisions.Related policy-gradient methods reduce variance by contrasting sampled trajectories with references.
  • Multi-Agent Learning and Verification: MASIRL uses a Grounder to generate candidate temporal segments and a frozen Verifier to score them for rewards, pseudo-target selection, and update gating.Gradients propagate only through the Grounder.
  • Multi-Agent Learning and Verification: The Verifier scores candidate segments with a query-conditioned log-likelihood ratio passed through a logistic sigmoid, without receiving gradients.Explicit segment boundaries focus scoring on the proposed interval rather than the full video.

B. Multi-Agent Group-Relative REINFORCE

MASIRL converts verifier judgments into group-relative policy-gradient rewards for sampled video-reasoning trajectories. The frozen Verifier selects representative evidence, supports reward construction, and coordinates trajectory reinforcement with group-level update gating.

  • Reward Definition and Carrier Selection: For each sampled completion, the Grounder produces candidate evidence segments, and the Verifier identifies a valid reward-carrier span.The selected span provides sequence-level credit for the trajectory.
  • Reward Definition: Trajectory rewards combine localization overlap, validity indicators, and Verifier quality scores when annotations are available.Without annotations, the IoU component is removed and the Verifier supplies the quality signal.
  • Group-Relative Advantage and GRR Loss: Group-relative advantages subtract each trajectory’s leave-one-out reward baseline, making updates reflect relative quality among sampled peers.This normalization reduces sensitivity to global reward scale and lowers estimator variance.
  • Group-Relative Advantage and GRR Loss: Positive advantages reinforce trajectories through lower negative log-likelihood, whereas negative advantages suppress them.The resulting GRR objective applies sequence-level credit to Grounder trajectories.
  • Verifier-Guided Training Procedure: The Verifier is reused to select reward carriers, rank calibration candidates, and gate updates for already-strong groups.This separates sequence-level credit assignment, boundary refinement, and difficulty-aware update selection within one training loop.
  • Verifier-Guided Training Procedure: After warmup, groups whose best candidate exceeds the current Verifier threshold have their advantages zeroed before the GRR update.The procedure focuses optimization on groups whose strongest proposals remain below the threshold.

C. Bootstrapped Boundary Calibration

Bootstrapped calibration supplies direct temporal-boundary supervision from verifier-ranked candidates. It uses multiple relevance-weighted pseudo-targets while stopping gradients through candidate selection and Verifier scores.

  • Calibration Targets: GRR updates language trajectories but does not directly regress temporal boundaries, so calibration trains the shared temporal head separately.The pseudo-targets come from the same candidate distribution that produced the policy-gradient signal.
  • Calibration Targets: The calibration target set contains the Top-Kcal candidates ranked by Verifier relevance across all sampled trajectories.If fewer candidates exist, the full candidate set is used.
  • Calibration Targets: Verifier-ranked pseudo-targets are selected by relevance rather than ground-truth IoU, and their weights are temperature-smoothed.The selection and weights are treated as constants, so gradients do not flow through the Verifier.
  • Calibration Loss: The calibration loss applies a boundary loss to the Grounder’s temporal prediction using several high-scoring candidates.Kcal>1 and finite temperature distribute supervision across spans instead of committing to one.
  • Calibration Loss: Reward-carrier selection reinforces one representative segment, whereas calibration averages supervision over multiple high-confidence spans.Stop-gradient treatment encourages the Grounder to improve its own temporal prediction rather than exploit the Verifier score surface.

D. Verifier-Based Update Gating

Verifier-based gating restricts policy-gradient updates to groups whose best candidate remains below a moving Verifier threshold. This focuses optimization on harder groups after an initial warmup.

  • Update Gating: After warmup, the method derives a threshold from smoothed recent validation Verifier scores and evaluates each group’s best candidate against it.Groups are selected for updates according to whether their strongest candidate remains below the current threshold.
  • Update Gating: Advantages are zeroed for already-strong groups, preserving the remaining hard groups’ relative advantage signs and scales.The method therefore avoids rescaling all trajectories within a batch when easy groups are excluded.

E. Full Objective and Training

The full training objective combines group-relative policy optimization with verifier-guided temporal calibration. Verifier scores operate at three granularities: trajectory credit, multi-candidate boundary supervision, and update gating.

  • Full Objective: The optimization objective is L(θ) = LGRR(θ) + λ Lcal(θ), combining policy-gradient and calibration losses.The coefficient λ is nonnegative.
  • Full Objective: LGRR updates language logits through advantage-weighted negative log-likelihood, while Lcal updates the shared temporal head through boundary regression or classification.Under weak- or no-label conditions, calibration provides a differentiable multiagent signal for temporal refinement.
  • Verifier Roles: Verifier scores select a reward carrier, define the Top-Kcal pseudo-target pool, and gate which groups receive policy-gradient updates.These roles coordinate sequence-level optimization and temporal refinement without conflating their purposes.

IV. EXPERIMENTS

The experiments test whether multi-agent training improves zero-shot transfer over a same-scale baseline across grounding and video QA benchmarks, and assess the effects of the framework’s training components.

  • The evaluation compares multi-agent training against a same-scale baseline under a zero-shot protocol without target-dataset fine-tuning.
  • The experiments cover temporal grounding, grounded video question answering, and general video question answering.
  • Component ablations examine how the framework’s three training components affect the reported results.

A. Experimental Setup

The setup evaluates a fixed 2B Grounder–Verifier system through zero-shot transfer across temporal grounding, grounded QA, and long-video QA. Results show modest, consistent gains over the same-scale baseline, strongest on relevance-oriented metrics rather than exact boundaries.

  • Datasets: Evaluation spans Charades-STA and ActivityNet-Captions for temporal grounding, NExT-GQA for grounded QA, and Video-MME, MLVU, and LVBench for general video QA.
  • Implementation Details: The Grounder is a 2B-parameter multimodal model, while a separately trained frozen 2B Verifier scores candidate segment quality.
  • Transfer Protocol: All target-benchmark results use zero-shot transfer without target-dataset fine-tuning, with verification examples disjoint from evaluation videos and samples.
  • Grounded Video QA: 28.7 mIoU and 25.4% Acc@GQA are achieved on NExT-GQA, with modest gains over the same-scale VideoMind baseline.
  • Cross-Benchmark Pattern: Across benchmarks, gains are clearest for mIoU, moderate-overlap recall, and answer accuracy, while strict boundary precision remains weaker.
  • General Video QA: 59.1% on MLVU, 35.6% on LVBench, and 54.1% on Video-MME exceed the same-scale baseline, with stable but small margins.
  • Temporal Grounding: 46.1 mIoU is achieved on Charades-STA, alongside 30.5 mIoU on ActivityNet-Captions under zero-shot temporal grounding.

C. Cross-task Consistency and Case Studies

Across benchmarks, MASIRL’s clearest gains occur on relevance-oriented metrics, while ablations show contributions from GRR, calibration, gating, and verifier-based reward selection.

  • Cross-task consistency: mIoU, R@0.3, R@0.5, and answer accuracy show the clearest gains, whereas R@0.7 changes little.This pattern is consistent with verifier scoring that emphasizes segment relevance rather than exact endpoint precision.
  • Component ablations: Removing GRR causes the largest single-component drop: −0.7 mIoU and −1.4% Acc@GQA on NExT-GQA, plus −1.3% on both Video-MME splits.The ablation weakens both grounding and QA metrics.
  • Component ablations: Setting λ=0 reduces NExT-GQA to 28.3 mIoU and 24.9% Acc@GQA, or −0.4 mIoU and −0.5% Acc@GQA versus the full model.The change is consistent with calibration influencing the temporal head while also affecting QA behavior.
  • Component ablations: Removing gating reduces NExT-GQA to 28.3 mIoU / 25.0% Acc@GQA and Video-MME to 53.6% / 45.4%.The reported setting supports gating as helpful, without establishing when its benefit is largest during training.
  • Sensitivity analysis: λ=0.3 is best among tested values, while increasing G from 2 to 4 improves both metrics and G=8 changes them only slightly at higher sampling cost.The sweep therefore favors λ=0.3 and G=4 within the tested grid.
  • Reward-carrier selection: Top-Kr reward-carrier selection exceeds first-valid by +0.8 R@0.5 and +0.6 mIoU, although all three strategies perform comparably.Its advantage is smaller than the main component-ablation drops.

E. Qualitative Analysis

Qualitative cases show MASIRL selecting tighter action-bearing intervals than baselines that include related preparatory or surrounding context, while recurring failures remain possible.

  • Case 1: In the cutting-board query, MASIRL closely matches the ground-truth interval and excludes the preceding cabinet-opening action.Several baselines begin too early, while VideoMind retains about 2 s of lead-in context.
  • Interpretation: The examples are qualitatively consistent with relevance-oriented gains and compatible with verifier-based preference over candidate spans.They remain illustrative rather than direct causal tests of the internal decision process.
  • Case 2: In the metal-bar query, MASIRL narrows the prediction to the specific showing gesture, achieving higher IoU with the ground truth.Baselines tend to cover the broader handling sequence.
  • Common failure patterns: A recurring failure retrieves the correct action but extends 3–5 s beyond the annotated endpoint.The qualitative analysis identifies over-inclusive temporal boundaries as a remaining issue.

F. Boundary Preference and Error Taxonomy

The verifier’s relevance-oriented objective favors segments containing the target action but provides weaker pressure for fine-grained temporal boundaries.

  • Error taxonomy: Early starts, late ends, and context over-inclusion arise when semantically compatible neighboring frames remain in the selected segment.These errors can preserve R@0.3 while degrading stricter metrics such as R@0.5, R@0.7, and mIoU.
  • Boundary preference: The verifier may score an extended segment highly whenever it contains the target action, regardless of extraneous preparatory or trailing context.This links the observed boundary errors to the verifier’s optimization target.
  • Future direction: Boundary-sensitive negatives are proposed as a future way to prefer tighter annotations and sharpen calibration targets and GRR rewards.The proposal remains a hypothesis rather than an established result.
  • Boundary preference: The main residual challenge is distinguishing neighboring spans that are semantically relevant but differ in boundary precision.The limitation is tied to relevance-based verification rather than evidence selection alone.

VI. CONCLUSION

The paper studies whether frozen verification can train a video Grounder rather than only rerank candidates at inference. Within the tested zero-shot setting, MASIRL consistently improves a same-scale baseline, especially on relevance-oriented metrics, but remains limited for strict boundary precision and broader claims.

  • Conclusion: MASIRL trains a Grounder with frozen verification, reusing candidate-span scores in GRR, calibration, and gating.The Grounder proposes evidence spans and answer-bearing trajectories, while the Verifier supplies the preference signal.
  • Conclusion: Across grounded QA, temporal grounding, and long-video QA, the design improves a same-scale 2B baseline most clearly on mIoU, moderate-overlap recall, and related relevance measures.The conclusion frames the result as consistent within the evaluated transfer setting.
  • Scope and limitations: The method is better viewed as a relevance-oriented training strategy for video evidence selection than as a complete solution to fine-grained temporal localization.Strict boundary precision improves less than relevance-oriented measures.
  • Future direction: Boundary-aware frozen verification is identified as a natural next step for distinguishing tight spans from over-inclusive ones.The paper presents this as future work rather than a demonstrated improvement.
Loading 2608.28675v1…