PaperlayerView exact revision on arXiv

Source-linked AI summary

Confidence-Aware Tool Orchestration for Robust Video Understanding

Yangfan He, Yujin Choi, Jaehong Yoon

arXiv:2606.26904v1cs.CVcs.AI

TL;DR

Video-LLMs can lose 15-30%p accuracy under realistic corruption while their confidence remains largely unchanged. Robust-TO integrates per-frame reliability into frame selection, tool routing, evidence synthesis, and GRPO training, achieving strong clean and corrupted-video results with the smallest clean-to-corrupted drop. Its setting assumes unknown, frame-varying corruption types and severities.

  • Problem

    Video-LLMs often treat frames as equally reliable, producing 15-30%p accuracy losses under realistic corruptions while self-reported confidence remains largely unchanged.

  • Method

    Robust-TO profiles frame quality, selects frames by reliability and query relevance, routes atomic sub-queries to corruption-matched tools, and synthesizes calibrated evidence with confidence-cost GRPO.

  • Results

    Robust-TO achieves 56.4% average accuracy on clean inputs and 54.3% on corrupted inputs, outperforming the strongest open-source baselines and showing the smallest clean-to-corrupted accuracy drop.

  • Takeaways & Limitations

    Reliability-aware adaptive tool use improves video reasoning across clean and corrupted benchmarks while reducing unnecessary frame processing and inference time.

  • Takeaways & Limitations

    The problem setting assumes corruption types and per-frame severity schedules are unknown in advance, with different frames potentially affected by distinct disturbances.

Abstract

from arXiv · show

Video reasoning language models implicitly assume that every input frame is equally reliable. This leads to what we term the Blind Trust Problem: under realistic perturbations such as motion blur, glare, or occlusion, frontier video reasoning models can suffer 15-30%p accuracy drops on real-world embodied benchmarks, while remaining unaware that their visual evidence has been degraded. To address this challenge, we propose Robust-TO, an agentic video understanding framework that explicitly integrates per-frame trustworthiness into every stage of reasoning. Robust-TO organizes heterogeneous visual perception tools under a unified evidence interface. Each tool receives a sub-query derived from the original question and a set of trustworthy frames selected by the reliability-relevance score. It returns evidence in a shared format: a concrete prediction (e.g., a bounding box, motion trajectory, recognized text, or action label), temporal grounding, and a calibrated reliability score. During reasoning, these calibrated scores guide evidence weighting in a three-tier synthesis process (high/medium/low) and define a confidence-cost GRPO reward that jointly optimizes correctness, evidence reliability, and efficiency. On two video reasoning benchmarks spanning eight tasks, Robust-TO achieves 56.4% average accuracy on clean inputs, surpassing the strongest open-source baseline by 10.6%p and outperforming Gemini-2.5-Pro (46.2%). Under five realistic corruption types, Robust-TO maintains 54.3% average accuracy, 5.8%p above the strongest open-source baseline, while exhibiting the smallest clean-to-corrupted accuracy drop among all compared methods.

1 Introduction

Robust-TO addresses the Blind Trust Problem by selecting trustworthy, query-relevant frames, routing sub-queries to visual tools, and weighting evidence by calibrated confidence. Across clean and corrupted videos, it improves accuracy while reducing processed frames and inference time.

  • Motivation: 15-30%p accuracy drops occur under common corruptions while frontier video reasoning models’ self-reported confidence remains largely unchanged.This silent failure motivates reliability-aware video reasoning in safety-critical settings.
  • Robust-TO: Robust-TO profiles frame degradation, ranks frames by reliability and query relevance, and retains the top-K trustworthy frames for downstream perception.Its quality profiling considers blur, brightness deviation, and occlusion, while filtering corrupted yet query-relevant distractors.
  • Robust-TO: Robust-TO decomposes input questions into atomic sub-queries and routes each to the perception tool best suited to the selected frames’ dominant corruption.Each tool call returns a result-confidence pair for fine-grained perceptual evidence.
  • Training: GRPO combines correctness, confidence-cost, sub-query efficiency, and format rewards to optimize accuracy, reliability, and tool-use parsimony.The confidence-cost reward penalizes expensive tool calls on degraded frames, while the efficiency reward penalizes over- and under-decomposition.
  • Results: 56.4% average accuracy is achieved on clean benchmarks, surpassing Gemini-2.5-Pro at 46.2% and supervised fine-tuned Qwen2.5-VL-7B at 45.8%.The evaluation spans UrbanVideo-Bench and VSI-Bench across eight tasks.
  • Results: 54.3% average accuracy is achieved on corrupted UrbanVideo-Bench, exceeding Video-R1 by 5.8%p and reducing processed frames by 35% while cutting inference time by over 35%.The adaptive key-frame selector reduced average processed frames from 32 to 20.7 while improving accuracy by 1.6%p.

2 Related Work

Prior work established video-language modeling, iterative tool use, and reinforcement learning for multimodal reasoning. Robust-TO builds on these directions by treating reliability as an explicit property of tool-produced evidence.

  • Video Large Language Models: Video-LLMs extend image-language pretraining to video by modeling temporal information through dynamic sampling, window attention, time embeddings, or shared image-video representations.These models achieve strong results across benchmarks including Video-MME, MVBench, and EgoSchema.
  • Agentic and Tool-Augmented Video Reasoning: Agentic video systems iteratively select frames and gather evidence using controllers, retrievers, captioners, memory, or entity-relation graphs.The cited systems include VideoAgent, a memory-augmented variant, and Graph-VideoAgent.
  • Agentic and Tool-Augmented Video Reasoning: Existing tool interfaces report what was found but not how reliably it was found.For example, an LLM controller calling a CLIP retriever cannot assess the reliability of the retrieved visual evidence from the interface alone.
  • Reinforcement Learning for Multimodal Reasoning: GRPO provides a memory-efficient alternative to PPO by estimating advantages from group-normalized rewards without a value network.Video-R1 extends GRPO with a temporal-order auxiliary reward, while DeepSeek-R1 demonstrates rule-based reflection and verification.

3 Robust-TO: Robust Video Understanding with Tool Orchestration

Robust-TO addresses unreliable visual evidence by estimating frame quality, selecting trustworthy and query-relevant frames, routing sub-queries to suitable tools, and integrating confidence-aware evidence.

  • Problem Setting: Existing video reasoning methods can admit corrupted frames because they often treat frames as equally reliable or select them by query similarity alone.Robust-TO instead combines quality profiling with query similarity and reports confidence alongside answers.
  • Problem Setting: Robust-TO models corrupted videos as streams whose frames may suffer unknown, heterogeneous degradations with varying severity.The stated examples include motion blur, glare, and partial occlusion.
  • Frame Selection via Quality Profiling: The framework first profiles blur, brightness deviation, and occlusion to estimate each frame’s disturbance and identify its dominant corruption and severity.Quality profiling uses signal-level statistics without training data or reference frames.
  • Frame Selection via Quality Profiling: A reliability-relevance score ranks frames by combining estimated reliability, 1 −d(fi), with query relevance measured by cosine similarity.The controller returns top-K valid frames, with K adaptively chosen from 4 to 12 according to query complexity.
  • Adaptive Tool Use: The host VLM decomposes questions into atomic sub-queries and routes each to perception tools selected by sub-query type and dominant corruption.Spatial queries can use detection tools, while temporal queries can use tracking or action-recognition tools.
  • Reliability-Aware Evidence: Each tool call returns a shared result-confidence pair, while evidence integration uses frame reliability to reduce overconfident reasoning from degraded inputs.Evidence records the tool output, confidence, source frames, and their disturbance values; the confidence is adjusted using conservative input reliability.
  • GRPO Training: The sub-query efficiency reward penalizes both insufficient and excessive decomposition relative to a question-dependent target count.The target count is estimated once per question by a separate frozen VLM.

4 Experiments

Experiments show that Robust-TO improves clean and corrupted-video reasoning through reliability-aware frame selection, collaborative evidence synthesis, confidence reporting, and adaptive tool routing. Ablations attribute gains to these components, while case studies illustrate resistance to corrupted evidence.

  • Main Results: 56.4% average accuracy is achieved on clean benchmarks with Qwen3-VL-7B, with the best performance on 6 of 8 tasks.The largest clean-video gains occur on temporally extended tasks such as Appearance Order and Landmark Position.
  • Main Results: 54.3% average accuracy is achieved on corrupted UV-Bench, leading every corruption type and producing the smallest clean-to-corrupted drop among methods.Robust-TO also outperforms Video-R1 at 48.5% and Gemini-2.5-Pro at 38.1%.
  • Ablation Studies: GRPO confidence-cost reward contributes the largest reasoning-paradigm gain at +4.7%p, while removing confidence-cost reward reduces accuracy by 2.3 points.Removing sub-query efficiency reward loses 2.1 points, and learning rather than freezing the sub-query estimator loses 1.2 points while making the reward 2.3 times more unstable.
  • Ablation Studies: The full confidence-guided policy outperforms query-only routing by +6.1%p and fixed-tool routing by +10.4%p on average.Gains are task-dependent, including +9.6%p on Appearance Order and +1.7%p on Route Planning.
  • Ablation Studies: The key-frame extractor reduces processed frames from 32 to 20.7 while improving accuracy from 49.1% to 50.7%.Training time decreases by over 16 hours and inference time per sample drops by more than 86 seconds.
  • Ablation Studies: Removing frame-quality coupling reduces accuracy from 50.7% to 43.1%, while replacing worst-K aggregation with a uniform mean causes a 3.3%p drop.The results indicate that clean frames should not mask unreliable evidence.

5 Conclusion

Robust-TO addresses the Blind Trust Problem by combining reliability-aware evidence handling with efficiency-aware training. The framework reports strong benchmark performance and identifies remaining scope limitations.

  • Robust-TO couples tool certainty with parameter-free disturbance estimates, reliability-ranked frame selection, corruption-matched routing, and three-tier evidence synthesis.
  • Robust-TO uses a confidence-cost GRPO reward with frozen-estimator sub-query efficiency to jointly optimize accuracy, reliability, and parsimony.
  • Robust-TO substantially outperforms the strongest open-source baseline across multiple benchmarks and tasks while achieving high clean accuracy with minimal clean-to-corrupted drop.
  • The framework’s limitations include a disturbance vocabulary restricted to blur, brightness, and occlusion, plus decomposition quality bounded by the frozen estimator.

A Limitations and Broader Impact

Robust-TO’s current disturbance profile covers common visual degradations but excludes several additional perturbation channels. Its explicit reliability signal is intended to support more accountable video reasoning in safety-relevant applications.

  • Limitations: The disturbance profile covers blur, brightness deviation, and occlusion, but excludes adversarial perturbations, semantic occlusions, and audio–visual misalignment.The authors describe extending the disturbance profile to additional channels as compatible with the framework’s plug-and-play interface.
  • Broader Impact: Robust-TO targets safety-relevant uses including forensic video analysis, surveillance review, and post-hoc autonomous-driving analysis.These settings involve consequences when degraded visual evidence produces silent failures.
  • Broader Impact: Per-frame reliability is designed as an explicit, interpretable signal for understanding why a conclusion was reached and how trustworthy its evidence is.The stated aim is to promote more accountable video reasoning systems.

B Details of Parameter Setting

Robust-TO consolidates parameter choices by pipeline stage, including disturbance constants, adaptive frame-selection settings, confidence synthesis, reward weighting, and training configuration. The reference tables document these settings, their sources, and available sensitivity information.

  • Disturbance and selection: The three disturbance channels use τblur=500, µref=0.5, and τedge=30 as signal-level constants calibrated once on a small held-out set.The disturbance channels are otherwise described as parameter-free.
  • Disturbance and selection: Frame selection uses (θrel, θsim)=(0.55, 0.30), while the host VLM adaptively chooses K in [4] [12].The reported average K is 7.8 on clean inputs and 6.2 on corrupted inputs.
  • Confidence and synthesis: Confidence synthesis uses a worst-K mean with K=⌈n/3⌉, clips intrinsic confidences to [0.01, 1.0], and groups evidence into high, medium, and low tiers.The high tier requires cj ≥0.7 and d<0.3, while the low tier uses cj <0.3 or d≥0.7.
  • Reward: The GRPO reward sets λ=0.5, α=0.2, and β=1.0, with auxiliary reward weight w=1/3.The total reward combines accuracy, sub-question, confidence, and format components.
  • Training: Training uses 4×A100-80GB GPUs, peak learning rate 1×10^-6, rollout group size 16, KL penalty 0.01, and approximately 5,000 steps.The configuration uses DeepSpeed ZeRO-2+FSDP, per-GPU batch 2, gradient accumulation 4, and a 200-step cosine warmup.
  • Reference tables: Table 8 organizes parameters by pipeline stage and reports sources plus measured sensitivity, while Table 9 defines tool costs and the shared interface.Table 9 calibrates normalized wall-time against caption_frame and permits user-defined tool sets with arbitrary costs.

B.1 Tool implementation details.

Robust-TO wraps pretrained visual tools behind a unified result-and-confidence contract. Each tool derives intrinsic confidence from a task-specific signal, which is clipped before producing final confidence values.

  • Unified interface: Every tool exposes the unified (result, confidence) contract, while the intrinsic confidence calculation depends on the tool.The library includes frame assessment, selection, detection, captioning, tracking, action recognition, and text recognition tools.
  • Selection and detection: assess_quality outputs the disturbance score with intrinsic confidence 1.0, whereas select_frames and retrieve_frames use frame-selection score and embedding similarity.The selection tools’ confidence is based on cosine similarity between frame and query embeddings.
  • Selection and detection: detect_objects uses mean GroundingDINO-T detection confidence after discarding boxes below 0.3.Its intrinsic confidence is averaged over the returned bounding boxes.
  • Task-specific confidence: caption_frame, track_temporal, recognize_action, and read_text derive confidence from token likelihood, tracklet IoU, action probability, and character recognition signals.These tools wrap the host VLM, ByteTrack, VideoMAE-v2, and PaddleOCR, respectively.
  • Final confidence: All intrinsic scores are clipped to [0.01, 1.0], after which final confidence cj is computed by multiplying intrinsic confidence with the framework’s reliability term.The clipping prevents zero-multiplication in Eq. (4).

B.2 Experiment Configuration

The experiment configuration uses GRPO training across both Qwen-VL backbones, with shared hyperparameters unless otherwise noted. Training uses the video subset of Video-R1 without additional video data or synthetic corruption augmentation.

  • GRPO training uses the same configuration for Qwen2.5-VL-7B and Qwen3-VL-7B unless otherwise noted.Training hyperparameters are summarized in Table 10.
  • Training uses approximately 12K video–question–answer triplets from Video-R1’s video subset.The data spans indoor navigation, outdoor driving, egocentric activities, and surveillance footage.
  • The training set excludes additional video data and synthetic corruption augmentation.Corruption robustness is acquired through the confidence-guided pipeline and confidence-cost reward.

B.3 Dataset and Evaluation Details

Evaluation spans two embodied video benchmarks with four tasks each, using accuracy on official splits. Robustness is tested with intermittent medium-severity corruptions, while the method section defines per-frame disturbance components and the adaptive-versus-uniform comparison.

  • Benchmarks: UrbanVideo-Bench evaluates outdoor embodied spatial reasoning across Landmark Position, Counterfactual, Progress Evaluation, and Action Generation.All tasks are multiple-choice, and results use accuracy on the official evaluation split.
  • Benchmarks: VSI-Bench evaluates indoor spatial intelligence through Relative Distance, Relative Direction, Route Planning, and Appearance Order.Its videos come from ego-centric indoor navigation, with accuracy reported under the official protocol.
  • Corruption evaluation: Robustness variants apply Motion Blur, Gaussian Noise, Glare, Occlusion, and Low-Light at medium severity to randomly selected 40–60% of video frames.The setup models intermittent degradation, such as a dashcam intermittently encountering glare from oncoming headlights.
  • Disturbance scoring: Per-frame disturbance scores combine blur, brightness, and occlusion components after min–max normalization across each video.The components quantify Laplacian-based blur, luminance deviation, and missing informative edge structure.
  • Disturbance scoring: Sharp frames have lower blur disturbance, extreme luminance increases brightness disturbance, and large uniform regions increase occlusion disturbance.Occlusion disturbance is inferred from fewer edge pixels using Sobel-magnitude statistics and threshold τedge=30.
  • Frame-selection comparison: Table 12 compares adaptive key-frame selection averaging 20.7 frames with uniform sampling of 21 frames under the same full tool-augmented reasoning pipeline.The comparison isolates frame-selection strategy as the difference between settings.

D Additional Experiments

At a matched frame budget and shared tool interface, adaptive frame selection outperforms uniform sampling.

  • 2.0%p higher average accuracy comes from adaptive selection than uniform sampling with Qwen2.5-VL-7B, 50.7 versus 48.7, using approximately 21 frames.Both settings use the full tool interface and consume approximately the same number of frames.

D.2 Ablation Studies

The ablations show that trustworthy-frame selection, frame-grounded query decomposition, confidence-guided routing, and reliability aggregation improve accuracy or robustness while reducing unnecessary processing. These benefits extend across corruption settings and backbone architectures, although several detailed comparisons are only partially reported in the supplied passages.

  • Sub-Query Decomposition: 3.8%p versus 5.0%p clean-to-corrupted accuracy drop favors Text+Frame over Text decomposition, because visual grounding better aligns sub-queries with selected trustworthy frames.Text+Frame additionally conditions decomposition on the visual content of the top-K selected frames.
  • Tool Routing: Confidence-guided routing adds frame quality to semantic tool selection, whereas query-only routing selects tools from sub-query type and fixed routing always uses caption_frame.The fixed-tool baseline is described as a lower bound because captioning is broadly applicable but least specialized.
  • Frame Selection: 3.6%p higher corrupted accuracy results from reliability-aware frame selection, which also reduces average forwarded frames from 32 to 20.7.Clean accuracy increases by 1.6%p, from 49.1 to 50.7, while discarding frames with high disturbance scores.
  • Reliability Aggregation and Estimation: Uniform-mean reliability aggregation yields the lowest clean and corrupted accuracy, while disturbance-estimator comparisons favor the parameter-free estimator over NIQE and BRISQUE.The estimator advantage spans all five corruption types, with especially large reported margins for Motion Blur and Occlusion.
  • Key-Frame Extraction: 35% fewer frames accompany Robust-TO’s key-frame extraction while gains over Video-R1 reach +7.7%p on clean data and +7.1%p on corrupted data with Qwen2.5-VL-7B.The supplied passage also reports consistent gains with the stronger Qwen3-VL-7B backbone.
  • Robustness: 2.3%p and 2.7%p corrupted drops for Video-R1 contrast with Robust-TO’s 2.9%p and 2.3%p drops, demonstrating more graceful degradation under the reported comparison.Video-R1 processes all frames at 1 fps without quality-based filtering.

D.3 Additional Case Studies

The section presents four corrupted-video case studies across UrbanVideo-Bench and VSI-Bench, then describes Robust-TO’s computational-budget analysis, soft routing preferences, and prompt templates for decomposing and routing sub-queries.

  • Additional Case Studies: Four additional case studies span UrbanVideo-Bench Landmark Position and Action Generation, plus VSI-Bench Relative Distance and Appearance Order, with varied task structures, video lengths, and corruption profiles.The cases cover urban drone flyover, drone delivery, kitchen walkthrough, and living-room sweep videos.
  • Additional Case Studies: The evaluation includes landmark-order and action-generation cases involving simultaneous blur, glare, occlusion, Gaussian noise, and motion-blur disturbances.The landmark case uses a 30-frame urban drone flyover, while the action-generation case uses a 20-frame drone approach toward power lines.
  • Relative Distance: Robust-TO correctly identifies the kitchen sink as closest to the refrigerator with MED-tier confidence, reflecting estimator noise in a near-miss.The standard VLM selects the trash bin, while the sink shares the refrigerator’s wall and counter.
  • Appearance Order: During the living-room appearance-order case, glare washes out red book spines, causing the standard VLM to invert the bookshelf and television ordering.The model reports the television before the bookshelf after breaking a detection tie by visual salience.
  • Computational Budget: Quality assessment and frame selection occur exactly once per question in clean and corrupted settings, while corruption changes average tool invocation patterns.Table 22 reports reduced detect_objects calls under corruption, from 1.42 to 0.87 per question.
  • Confidence-Guided Tool Routing: The pipeline decomposes each complex question into a minimal set of atomic sub-queries, assigns semantic types, and routes each sub-query to a tool based on corruption conditions.Routing is soft and selected by the host VLM through in-context reasoning rather than a hard-coded lookup table.
Loading 2606.26904v1…