PaperlayerView exact revision on arXiv

Source-linked AI summary

ParaVT: Taming the Tool Prior Paradox for Parallel Tool Use in Agentic Video Reinforcement Learning

Zuhao Yang, Kaichen Zhang, Sudong Wang, Keming Wu, Zhongyu Yang, Bo Li, Xiaojuan Qi, Shijian Lu, Xingxuan Li, Lidong Bing

arXiv:2605.20342v2cs.CV

TL;DR

Sequential video-tool calls limit long-video reasoning through error propagation and accumulating context. ParaVT enables parallel crop dispatch with PARA-GRPO, improving performance across long-video evaluation splits by an average relative gain of +7.9%.

  • Problem

    Existing native-RL methods dispatch one video-tool call per turn, limiting long-video reasoning through sequential error propagation and accumulating tool context.

  • Method

    ParaVT uses parallel multi-agent video-tool dispatch, while PARA-GRPO anchors structural parseability and gates tool incentives through randomized frame budgets.

  • Results

    +7.9% average relative gain across seven long-video evaluation splits over the Qwen3-VL-8B base model, with improvements on every split.

  • Takeaways & Limitations

    ParaVT demonstrates that anchoring format and gating tool incentives can provide a transferable recipe for agentic RL as tool capabilities become increasingly internalized.

  • Takeaways & Limitations

    The full PARA-GRPO pipeline has been validated only on Qwen3-VL-8B, leaving its generalization to other tool-native LMM families unresolved.

Abstract

from arXiv · show

Training large multimodal models (LMMs) via reinforcement learning (RL) to natively invoke video-processing tools (e.g., cropping) has become a promising route to long-video understanding. However, existing native-RL methods dispatch tool calls sequentially (i.e., one per turn): a single wrong crop propagates errors without peer correction, multi-turn tool calls corrupt context, and inference cost scales linearly with the number of turns. We introduce ParaVT, the first multi-agent end-to-end RL-trained framework for Parallel Video Tool calling, dispatching multiple time-window crops in a single turn for cleaner context and better fault tolerance. Yet applying standard RL to ParaVT reveals an obstacle we term the Tool Prior Paradox: the pretrained tool priors that enable tool exploration also destabilize cold-started structural format and expose the skip-tool reward shortcut under temperature sampling. A cross-model contrast on a weaker-prior LMM supports this claim: format stays stable but RL elicits zero tool calls, indicating that prior strength is the shared driver of both format collapse and tool exploration. We propose PARA-GRPO (Parseability-Anchored and Ratio-gAted GRPO), which augments standard RL with two complementary mechanisms: (i) a targeted format reward applied only at the structural-token positions most prone to collapse, and (ii) a per-prompt frame-budget randomization that creates training prompts where calling the tool yields a measurable reward signal over skipping it. Across six long-video understanding benchmarks, ParaVT improves over the Qwen3-VL baseline by +7.9% on average, with PARA-GRPO lifting training-time format compliance from 0.13 to 0.64. As tool capabilities become increasingly internalized in modern LMMs, RL must cooperate with the resulting priors, and ParaVT offers a general recipe for agentic RL. Code, data, and model weights are publicly available.

1 Introduction

ParaVT frames long-video understanding as agentic parallel tool use, addressing the Tool Prior Paradox in which stronger pretrained tool priors enable exploration but destabilize output format. PARA-GRPO combines parseability anchoring with frame-budget randomization to preserve valid tool calls and reward exploration.

  • ParaVT: ParaVT is an end-to-end RL framework for parallel video-tool calling that dispatches multiple temporal-window crops to parallel sub-agents and aggregates their evidence.Independent window grounding lets peer sub-agents outvote a single mis-localization while reallocating the visual budget across peers.
  • Tool Prior Paradox: Vanilla GRPO suffers Format Fragility: temperature sampling can revive the pretrained <tool_code> schema, corrupting parseable <think>/<tool_call>/<answer> trajectories and their reward signal.The SFT-learned format is reliable under greedy decoding but can collapse within a few vanilla-GRPO steps under temperature sampling.
  • Tool Prior Paradox: Qwen2.5-VL retains near-perfect format but emits no tool calls, whereas stronger-prior Qwen3-VL explores tool use while collapsing on format, exposing the Tool Prior Paradox.The contrast indicates that weakening pretrained tool priors stabilizes format but cancels tool exploration, while stronger priors expose the skip-tool shortcut.
  • PARA-GRPO: PARA-GRPO combines Exploration Anchoring and nFrames Gating to tame the paradox through selective structural-token rewards, constrained opening-tag generation, and randomized per-prompt frame budgets.The first mechanisms anchor rollout parseability without restricting reasoning content or tool-call sequences; frame-budget randomization creates prompts where tool use yields a measurable advantage over skipping.
  • Training setup: 97K SFT samples and a separate 4.4K RL split train ParaVT across general video QA, parallel-tool traces, long-video reasoning, open-ended QA, multiple-choice, and temporal grounding.The paper states that code, data, and model weights are publicly available.

2 Related Work

Prior long-video RL methods span tool-free, multi-agent, and single-LMM tool-augmented approaches, while ParaVT extends agentic tool-use research to video. The paper frames format stability as necessary for learning tool use and emphasizes context preservation because visual tokens dominate video rollouts.

  • RL for Long-Video Understanding: Long-video RL methods comprise tool-free reasoning, multi-agent cooperation, and single-LMM policies that emit structured tool calls during rollouts.The single-LMM branch includes LongVT with sequential crop_video calls and Zoom-Zero with a single coarse-to-fine zoom-in.
  • Format Stability and Tool Use in RL: Format stability is a precondition for tool-use learning because only parseable rollouts can receive credit for their tool calls.The paper relates its tool-native RL-time format fragility to the debated shallow-alignment intuition about early output tokens.
  • RL for Long-Video Understanding: Parallel tool calling dispatches K concurrent calls to independent sub-agents, enabling correct peers to outvote mis-localized crops and aggregate evidence.Sequential calling re-includes the full context across turns, accumulates visual-token overhead, and lets one bad crop amplify errors.
  • Format Stability and Tool Use in RL: In video tool-use research, context preservation is the primary design constraint because visual tokens dominate rollout context.This work extends the broader agentic-LLM tool-use literature to video rather than prioritizing token efficiency.

3 Method

ParaVT combines single-turn parallel video-tool dispatch with a cold-start SFT stage and verifiable-reward RL. PARA-GRPO addresses structural parseability and insufficient reward contrast for tool use through targeted boundary anchoring and prompt-level tool necessity.

  • Parallel-dispatch architecture: ParaVT emits K tool calls in one turn over disjoint windows, delegates them to weight-sharing sub-agents, and aggregates textual summaries into one tool response.Each sub-agent samples its assigned temporal crop and returns a summary rather than resampled frames.
  • Parallel-dispatch architecture: Parallel dispatch provides peer-correctable evidence, controlled context growth, and concurrent execution whose latency is bounded by the slowest sub-agent.Independent summaries can outvote a mis-localized window, while text responses add only a small constant context extension.
  • Two-stage training pipeline: The method cold-starts Qwen3-VL-8B-Instruct with supervised parallel-tool traces before GRPO, because the base model cannot natively produce parseable parallel calls in one turn.Without this exposure, probe RL fails to produce parseable rollouts and the GRPO advantage signal collapses before tool-use credit is assigned.
  • Dataset construction: The SFT corpus contains 97K samples across general video QA, long-video reasoning, temporal grounding, and self-curated parallel-tool traces.The parallel-tool component includes 15K LongVT rollouts, 5K Gemini-2.5-Flash distillations, and 2.5K MUSEG multi-segment grounding samples.
  • PARA-GRPO: PARA-GRPO also addresses the Tool Necessity Gap, where equal rewards for calling and skipping crop_video produce near-zero GRPO advantages for tool adoption.Anchoring restores parseability, but generous overview budgets can make both tool-calling and tool-skipping rollouts correct, eliminating the reinforcing gradient.
  • PARA-GRPO: PARA-GRPO combines constrained generation with selective structural anchoring to repair closing-tag collapse without restricting intermediate reasoning or tool-call content.The anchor rewards correct </think> closure and preserved <think>→</think>→<answer> structure, while penalizing an opened but unclosed <think>.

4 Experiments

Experiments evaluate ParaVT under a unified six-benchmark protocol and show gains over comparable open-source and proprietary baselines. Ablations attribute these improvements to PARA-GRPO’s joint stabilization of format compliance and tool use, while parallel dispatch improves accuracy without retraining.

  • Evaluation: ParaVT outperforms comparable open-source 7–8B baselines on six of seven evaluation splits and improves over Qwen3-VL-8B across every split.Evaluation uses six long-video benchmarks with a unified 64-frame adaptive protocol.
  • Evaluation: +7.9% is ParaVT’s average relative gain across all seven splits, including +15.7% on LongVideoBench, +20.2% on LVBench, and +11.5% on MLVU.ParaVT also reaches 50.1 mIoU on Charades-STA.
  • Evaluation: ParaVT surpasses GPT-4o on LVBench (39.8 vs. 34.7) and MMVU (68.6 vs. 66.7).These results close the reported open-source-to-proprietary gap on long-video reasoning.
  • Training Stage: Vanilla GRPO reverses the cold-started checkpoint’s tool overuse, dropping tool-call rate from κ=2.50 to κ=0.02 by skipping tools within 7 steps.The cold-started checkpoint imitates tool-using demonstrations, whereas vanilla GRPO exploits a reward shortcut.
  • Training Stage: PARA-GRPO reaches the highest Block A training-time format reward (fτ=0.41, κ=0.21) and strictly improves vanilla GRPO across all six evaluation splits.Its largest gains are on LongVideoBench and MMVU.
  • Component Effectiveness: Exploration Anchoring alone reaches fτ=0.35 and κ=0.19, while nFrames Gating alone reaches κ=1.36 with fτ=0.10; only the full recipe combines both objectives.The full recipe outperforms every per-component variant on all six evaluation splits.
  • Dispatch Mode: Parallel dispatch outperforms sequential dispatch on every tested benchmark, improving accuracy without retraining the same trained checkpoint.The largest gains occur on LongVideoBench and LVBench.

5 Conclusion · Appendix

ParaVT is presented as a multi-agent, end-to-end RL framework for parallel video-tool calls, with format anchoring and gated tool incentives addressing the Tool Prior Paradox. The appendix documents implementation, evaluation, rollout behavior, training dynamics, cross-model evidence, tool usage, and negative results.

  • 5 Conclusion: ParaVT enables tool-native LMMs to dispatch parallel video-tool calls in one turn for long-video reasoning, replacing sequential chains with peer-correctable evidence aggregation.The framework keeps inference cost flat as the number of dispatched tools grows.
  • 5 Conclusion: 97K-sample multi-task SFT corpus and 4,406-sample RL split span open-ended QA, multiple-choice, and temporal grounding.The RL split covers the listed task types and is separate from the SFT corpus.
  • 5 Conclusion: Six of seven long-video evaluation splits show ParaVT outperforming existing open-source 7–8B baselines.The result supports anchoring format and gating tool incentives as a transferable recipe for agentic RL.
  • Appendix: The appendix covers limitations and broader-impact scope limits and dual-use considerations.These topics are organized in Section A.
  • Appendix: Implementation details include hardware, SFT-data curation, sequential-to-parallel conversion, Gemini-CoT distillation, RL filtering, optimization, rewards, and token budgets.Section B also describes DAPO zero-gradient filtering and format/storage choices.
  • Appendix: Prompts and templates provide verbatim SFT, RL, and evaluation system prompts, including per-baseline-class evaluation prompt classes.These materials are collected in Section C.
  • Appendix: Sections D–G report rollout examples, training dynamics, cross-model evidence, and tool-call trajectories under three reward configurations.The appendix includes format-collapse mitigation examples, format–evaluation correlation, per-tag format closure, and before/after model trajectories.
  • Appendix: Section H groups negative results by reward-shape, data-shape, and gradient/format-shape intervention axes.Examples include phase-reward staging, Pre-RFT, stronger cold-start, Token-Decoupled GRPO structural masking, and bidirectional tag reversion.

A Limitations, Broader Impact, and Future Work

ParaVT’s main RL benefit is deployment-time format and tool-use stability, while its broader impact includes both beneficial long-video search and increased surveillance throughput. Future work targets scaling PARA-GRPO and extending necessity gating to other agentic settings.

  • Limitations: The RL stage primarily improves deployment-time format and tool-use stability under temperature sampling, not standalone greedy-evaluation performance over the cold-started checkpoint.Further amplifying the evaluation-time translation remains an open direction.
  • Limitations: Cross-model evidence for the prior’s role comes from a single Qwen2.5-VL versus Qwen3-VL contrast, which is consistent with causality but not a controlled intervention.
  • Broader Impact: Agentic long-video understanding can lower the human cost of searching extended footage for accessibility, sports analytics, and archival retrieval.
  • Broader Impact: ParaVT’s parallel-tool dispatch also reduces the marginal cost of large-scale CCTV or body-camera surveillance by amplifying throughput, motivating explicit consent and transparency frameworks.
  • Future Work: Future work will scale PARA-GRPO to 32B–72B LMMs and extend necessity gating to settings such as retrieval-augmented generation and code execution.

B Implementation Details · C Prompts and Templates

The implementation uses Qwen3-VL-8B-Instruct with capped 1-fps video inputs, disjoint SFT/RL data, and parallel-tool training infrastructure. Prompt templates enforce structured reasoning, parallel crop calls, and answer formatting, while parallel processing reduces visual-context re-encoding as tool-call counts grow.

  • B Implementation Details: Qwen3-VL-8B-Instruct is the base model, with videos decoded at 1 fps and uniformly subsampled to at most 64 frames.SFT and RL use max_pixels = 50176 (224×224), fps = 1, and max_frames = 64.
  • B Implementation Details: 30% of the 97K-sample SFT mix contains tool-bearing examples, outperforming an earlier 212K-sample mix with 14% tool-bearing data in downstream RL.The comparison motivates prioritizing the fraction of tool-bearing samples over absolute sample count once substantial non-tool video QA is present.
  • B Implementation Details: The SFT sources’ sequential LongVT-style traces are converted into single-turn parallel format by merging adjacent independent calls whose target windows do not overlap.Independence additionally requires that consumed tool responses contain no cross-reference to one another.
  • B Implementation Details: 5K Gemini-CoT traces use Gemini-2.5-Flash, with Qwen3-VL-235B fallback distillation for filtered pairs and GPT-4o cleanup for malformed JSON or invalid samples.Cleanup discards samples violating start_time < end_time or containing an empty answer field.
  • B Implementation Details: Parallel processing re-encodes visual context O(1) rather than O(K) times, where K is the number of tool calls required by a sample.A 64-frame overview consumes ≈16.4K visual tokens per turn under Qwen3-VL’s 256 visual tokens per frame.
  • B Implementation Details: At K=2.5, sequential input tokens are Tseq≈41K versus parallel Tpar≈16.5K, yielding a ∼60% reduction.The passage attributes the saving to the parallel architecture’s asymptotic context-reuse advantage.
  • C Prompts and Templates: The tool-augmented prompt directs the model to reason in <think>, issue multiple JSON-formatted crop_video <tool_call> blocks for different temporal windows, and answer in <answer>.It prohibits <tool_code>, Python syntax, other tool formats, and repeated temporal windows.
  • C Prompts and Templates: Non-tool sources use a minimal <think>...</think> then <answer>...</answer> prompt, while evaluation pins one prompt class per baseline.Direct-answer Qwen2.5-VL-Instruct receives no system prompt or required schema; reasoning baselines use the standard reasoning-and-answer scaffold.

D Rollout Examples

Three rollout examples illustrate Format Fragility and its mitigation: greedy SFT preserves native tool-call structure, early RL under temperature sampling collapses it, and PARA-GRPO restores it.

  • D Rollout Examples: The three examples are presented as representative demonstrations of the Format Fragility phenomenon and its mitigation.They contrast successful formatting, collapse, and recovery across SFT, early RL, and PARA-GRPO.
  • D Rollout Examples: SFT under greedy decoding produces well-formatted native output with two parallel crop_video calls and an answer identifying the cup pickup at 01:45.The rollout includes closed reasoning, two tool calls, tool responses, and a tagged answer.
  • D Rollout Examples: Under temperature sampling (τ=0.7) without PARA-GRPO, early RL causes format collapse, including an unclosed reasoning tag, non-native tool syntax, and no answer tag.The output reverts to a Python-style tool prior and is truncated after extended reasoning.
  • D Rollout Examples: With all PARA-GRPO components active, sampling preserves structured reasoning, two parallel crop_video calls, tool responses, and a tagged answer locating the pickup around 01:42.The recovered rollout explicitly explains that eight overview frames require targeted temporal crops.

E Training Dynamics … G Tool Usage Patterns

Training analysis separates SFT and RL contributions, while trajectory evidence shows PARA-GRPO stabilizes format and tool use across models. Format compliance recovers after vanilla GRPO collapse, and training-time format reward correlates with evaluation accuracy.

  • E Training Dynamics: Figure 5 decomposes the main evaluation gains into separate SFT and RL contributions.
  • F.1 Per-Tag Format Closure: Vanilla GRPO halves cold-start-learned closure rates within 9 steps as the policy reward-hacks toward direct answering.The rates are measured from raw τ=0.7 training-stream rollouts, reflecting compliance during RL exploration.
  • F.1 Per-Tag Format Closure: PARA-GRPO restores all three structural-tag closure rates above the cold-started step 500 baseline by step 19.
  • F.1 Per-Tag Format Closure: Selective Anchoring restores closure by applying the full PARA-GRPO recipe at structural-boundary tokens.
  • F Cross-Model Evidence: The cross-model analysis compares Qwen2.5-VL and Qwen3-VL at two endpoints of the tool-prior gradient and across the full 520-step training horizon.
  • F.2 Two-Model Trajectory: Training-time format reward correlates with greedy-eval VideoMME accuracy at Pearson r=0.86 (p<0.01) across 10 PARA-GRPO checkpoints.The cold-started step 500 checkpoint is marked as the pre-RL anchor.
  • F.2 Two-Model Trajectory: Qwen3-VL’s format rises from 0.13 to 0.41 under PARA-GRPO, while Qwen2.5-VL stays near 0.85 and emits zero tool calls.Qwen3-VL settles at κ=0.21 calls per rollout, contrasting with Qwen2.5-VL’s zero tool calls.
  • G Tool Usage Patterns: Figure 8 traces training-time tool-call trajectories under no penalty, no-tool penalty only, and full PARA-GRPO reward configurations.

H Negative Results and Failure Modes

The negative results are organized by reward-, data-, and gradient/format-shape interventions, each failing for a distinct reason that constrains the design space. Early GRPO exhibits reward hacking, while PARA-GRPO stabilizes tool-use control and improves format compliance during training.

  • Failure-mode axes: Negative results span reward-shape, data-shape, and gradient/format-shape interventions, including phase staging, task-aware coefficients, Pre-RFT, stronger cold-start, TD-GRPO masking, and bidirectional tag reversion.Each intervention fails for a distinct reason, further constraining the design space.
  • Reward-shape failures: 2.5 to 0: early GRPO tool usage collapses by step 7 under temperature sampling, indicating reward hacking.The rollout averages use τ=0.7 and group size 8.
  • Reward-shape failures: The no-penalty variant preserves tool calls but leaves format compliance low.This separates tool-call retention from structural-format recovery.
  • PARA-GRPO behavior: κ stabilizes at 0.1–0.5 under PARA-GRPO while fτ climbs to 0.41.These are training-time rollout averages measured at τ=0.7 with group size 8.

H.1 Reward-Shape Interventions

Reward-shape ablations show that format learning depends on accuracy feedback rather than succeeding through isolated staging. Task-aware coefficients improve training-time accuracy signals but do not improve held-out performance over base PARA-GRPO.

  • Phase Reward Staging: After 160 steps of format-only optimization, fτ remained at 0.13, indicating format and accuracy signals are interdependent.Accuracy gradients appear necessary to motivate initial format learning.
  • Task-Aware Reward Coefficients: Training-time accuracy reward improves from 0.15 to 0.24 over 30 steps with task-aware coefficients, while format compliance stays comparable at 0.39.The coefficients are 1.5× for concise MCQ answers and 0.3–0.4× for verbose answers.
  • Task-Aware Reward Coefficients: VideoMME reaches 61.81 and LongVideoBench reaches 58.26, below PARA-GRPO’s 62.11 and 60.40, respectively, on held-out evaluation.Task-aware shaping improves training signal quality without outperforming base PARA-GRPO.

H.2 Data-Shape Interventions · H.3 Gradient and Format-Shape Interventions

Data-shape expansions and rejection-based pre-RFT failed to improve RL: stronger cold starts eliminated tool exploration, while partially formatted data degraded downstream quality. Gradient masking and tag substitution likewise failed to stabilize format compliance, with fτ falling to 0.11 and Format Fragility persisting across tags.

  • H.2 Data-Shape Interventions: Pre-RFT peaked at fτ=0.40, but partially formatted SFT samples degraded cold-start quality on every downstream metric.The experiment sampled τ=0.7 rollouts, filtered format-compliant samples, and mixed them into SFT before RL.
  • H.2 Data-Shape Interventions: 12% parallel tool-calling augmentation strengthened VideoMME cold start from 61.3 to 62.3, yet RL from that initialization produced zero tool calls.The augmented dataset contained 106K samples versus 97K, while stronger SFT coverage shifted the policy toward reproducing the cold-start distribution.
  • H.2 Data-Shape Interventions: These failures motivated restricting cold-start training to the format schema rather than expanding it into the tool-call distribution.The reported factors included averaged-out delayed gradients, mixed single/parallel patterns increasing Format Fragility, and reduced exploration.
  • H.3 Gradient and Format-Shape Interventions: Token-Decoupled GRPO zeroed policy-gradient contributions from structural tokens such as <think> and <tool_call>, updating only semantic content tokens.The intervention followed prior sparse critical-token reweighting work but directly removed learning signals from format positions.
  • H.3 Gradient and Format-Shape Interventions: After 11 steps, fτ dropped to 0.11 below the 0.13 baseline, showing that masking format-token gradients made format irrelevant to reward.This outcome was opposite to the requirement for stabilizing format.
  • H.3 Gradient and Format-Shape Interventions: SFT with native <tool_call> used a single added tokenizer token, whereas non-native <tool_code> comprised four subwords encountered during pretraining.The tag substitution tested whether Format Fragility arose from mismatch between the SFT tag and the pretraining prior.
  • H.3 Gradient and Format-Shape Interventions: Tag substitution did not remove Format Fragility: RL emitted <tool_code> in 3.6–3.9% of <tool_call>-SFT rollouts and <tool_call> in 5.4% of <tool_code>-SFT rollouts.For <tool_code>-SFT, the SFT-trained tag appeared in only 1.8% of rollouts.
Loading 2605.20342v2…