PaperlayerView exact revision on arXiv

Source-linked AI summary

Are Video Reasoning Models Ready to Go Outside?

Yangfan He, Changgyu Boo, Jaehong Yoon

arXiv:2603.10652v3cs.CVcs.AI

TL;DR

Video-language models degrade under realistic weather, occlusion, lighting, and camera disturbances, exposing a gap between clean evaluation and real-world robustness. ROVA addresses this with consistency-based robust training and adaptive sample selection, while PVRBench evaluates perturbed video reasoning. Across multiple benchmarks, ROVA improves robustness and also transfers gains to clean videos, although its reward design and memory management retain stated limitations.

  • Problem

    Video-language models face realistic disturbances that substantially degrade understanding and reasoning, while standard evaluations mainly use clean, controlled conditions.

  • Method

    ROVA combines clean–corrupted consistency rewards with self-reflective, difficulty-aware online training that prioritizes informative samples, alongside the PVRBench perturbation benchmark.

  • Results

    ROVA consistently improves accuracy and reasoning quality across PVRBench, UrbanVideo, and VisBench, with gains extending to clean videos and larger variants matching or exceeding leading proprietary models.

  • Takeaways & Limitations

    ROVA provides a practical training recipe and benchmark for studying robustness to diverse realistic video perturbations while preserving improvements on clean video–question pairs.

  • Takeaways & Limitations

    The framework uses binary reward components, fixed equal reward weights, and an external LLM judge, while repeated re-evaluation can cause unbounded memory growth without threshold-based eviction.

Abstract

from arXiv · show

In real-world deployment, vision-language models often encounter disturbances such as weather, occlusion, and camera motion. Under such conditions, their understanding and reasoning degrade substantially, revealing a gap between clean, controlled (i.e., unperturbed) evaluation settings and real-world robustness. To address this limitation, we propose ROVA, a novel training framework that improves robustness by modeling a robustness-aware consistency reward under spatio-temporal corruptions. ROVA introduces a difficulty-aware online training strategy that prioritizes informative samples based on the model's evolving capability. Specifically, it continuously re-estimates sample difficulty via self-reflective evaluation, enabling adaptive training with a robustness-aware consistency reward. We also introduce PVRBench, a new benchmark that injects real-world perturbations into embodied video datasets to assess both accuracy and reasoning quality under realistic disturbances. We evaluate ROVA and baselines on PVRBench, UrbanVideo, and VisBench, where open-source and proprietary models suffer up to 35% and 28% drops in accuracy and reasoning under realistic perturbations. ROVA effectively mitigates performance degradation, boosting relative accuracy by at least 24% and reasoning by over 9% compared with baseline models (QWen2.5/3-VL, InternVL2.5, Embodied-R). These gains transfer to clean standard benchmarks, yielding consistent improvements.

1 Introduction

Current video-language models often fail under realistic disturbances, motivating ROVA’s robustness-aware training and PVRBench’s perturbation-based evaluation. ROVA improves accuracy and reasoning across perturbed and clean video benchmarks.

  • Realistic weather, occlusion, illumination, and camera-motion disturbances expose a gap between controlled evaluations and reliable real-world video reasoning.
  • ROVA generates corrupted videos, measures clean–corrupted reasoning and answer divergence, and selectively trains on moderately difficult samples.Overly easy samples are discarded, while excessively difficult samples are retained for later revisiting.
  • PVRBench injects 12 spatially aware and temporally coherent corruption styles across 27 scene categories to evaluate robustness under realistic disturbances.The benchmark covers lighting, camera motion, occlusion, and weather perturbations.
  • 11–17% and 10–14% drops affect proprietary models’ accuracy and reasoning, while open-source models lose up to 35% and 26%, respectively, under realistic perturbations.
  • ROVA outperforms models across PVRBench, UrbanVideo, and VisBench for accuracy and reasoning quality, with improvements extending to clean videos.Larger ROVA variants match or exceed leading proprietary models, and reasoning becomes more consistent under visual corruption.

2 Related Work

Prior work improves multimodal robustness through augmentation, adaptation, and transfer strategies, but existing video benchmarks and methods largely assume clean or generic perturbation settings. Structured, semantically grounded real-world video disturbances remain insufficiently addressed.

  • Existing multimodal robustness approaches use data augmentation, test-time adaptation, and transfer-based strategies for distribution shifts and adversarial inputs.
  • Most approaches target generic perturbations or optimization efficiency rather than structured, semantically grounded disturbances in real-world video.
  • Most video reasoning benchmarks evaluate nearly clean visual conditions, implicitly assuming stable lighting, unobstructed views, and smooth camera movement.

3 Training Robust Video Reasoning Models with ROVA

ROVA trains robust video reasoning through structured spatio-temporal corruption, adaptive sample selection, and consistency alignment between clean and perturbed inputs.

  • Structured Spatio-Temporal Corruption: ROVA generates corruption-augmented videos using realistic weather, lighting, occlusion, and camera-motion disturbances with spatial and temporal structure.It combines style-specific masks, temporal shuffling, and continuous per-pixel effect modulation.
  • Structured Spatio-Temporal Corruption: The corruption pipeline jointly disrupts temporal order and applies spatially continuous masks while preserving essential visual semantics.This design targets perturbation-invariant representations rather than independent pixel- or frame-level augmentation.
  • Difficulty-Aware Online Training: ROVA performs online self-reflective evaluation to classify samples as easy, difficult, or informative according to the model’s current responses on clean and corrupted inputs.The model also produces confidence scores, and the resulting policy adapts sample selection as capability evolves.
  • Difficulty-Aware Online Training: High-confidence easy samples are filtered, informative and low-confidence easy samples are trained immediately, and difficult samples are deferred in a temporal memory buffer.Stored samples are periodically re-evaluated, with difficult entries retained and easy entries removed from memory.
  • Dual-Branch Alignment: The dual-branch alignment mechanism optimizes the perturbed video branch against a clean branch using reasoning- and answer-aware rewards with GRPO.The clean branch acts as a fixed anchor while training enforces consistency between clean and perturbed outputs.

4 Evaluating Video Reasoning under Various Realistic Disturbances

PVRBench evaluates video reasoning under structured real-world disturbances rather than predominantly clean visual conditions. It combines realistic perturbation injection with metrics for both answer accuracy and reasoning reliability.

  • Motivation: Existing video reasoning benchmarks primarily evaluate clean visual conditions, whereas deployment involves adverse weather, occlusion, illumination changes, and camera instability.This contrast motivates evaluation beyond stable, unobstructed, and smoothly captured videos.
  • Benchmark Construction: PVRBench injects structured, semantically grounded perturbations into video reasoning data to test reliability under realistic visual variations.Its perturbations are conditioned on scene content, including depth layout, object locations, and motion patterns.
  • Evaluation Metrics: The benchmark reports accuracy for lighting, occlusion, camera shake, and weather, alongside Fragility, Consistency, Belief, Recovery, and Attention metrics.Reasoning metrics use a 0–5 scale, with higher values better except Fragility, and include clean-data averages for comparison.

5 Experiment

ROVA improves robustness and efficiency across perturbed and clean video benchmarks, while its difficulty-aware curriculum and structured masking provide measurable training and generalization benefits.

  • Main Results: 21–35% accuracy and 16–28% reasoning-quality drops affect most open-source video LLMs under perturbations relative to clean inputs.ROVA instead achieves consistent improvements in reasoning quality under visual corruption.
  • Main Results: ROVA improves answer accuracy and reasoning quality on PVRBench and unseen VisBench and UrbanVideo benchmarks under both clean and perturbed videos.The gains are reported without domain-specific fine-tuning and while maintaining strong clean-data performance.
  • Main Results: 5.9% fewer GPU-hours result from the SRE + DRE + ME curriculum compared with naive Dual-Branch training.The curriculum offsets the dual-branch design’s doubled forward-pass overhead.
  • Ablation Study and Analysis: 3.4% higher PVRBench accuracy than random discarding is achieved by difficulty-aware data selection across discard rates.The strategy increasingly reclassifies initially difficult samples as easy as training progresses and filters low-utility or noisy instances.
  • Ablation Study and Analysis: The reasoning reward provides the largest ablation gain, followed by easy-sample discarding; memory and temporal shuffle add smaller, consistent gains.The memory module and temporal shuffle are described as complementary regularizers that stabilize training and enhance robustness.
  • Ablation Study and Analysis: 6 - 9% absolute gains over fixed-shape and pixel-level random masking show that structured masks transfer to held-out perturbation types.The result supports broad robustness from a small subset of corruption mask styles.

6 Conclusion

ROVA combines structured spatio-temporal corruptions, dual-branch alignment, and self-reflective data curation, while PVRBench evaluates robustness under realistic disturbances. The framework improves performance on perturbed and clean video–question pairs and provides a benchmark and training recipe for future embodied reasoning research.

  • ROVA uses structured spatio-temporal corruptions, dual-branch alignment, and self-reflective data curation to learn perturbation-robust representations.
  • PVRBench evaluates embodied video reasoning robustness under realistic disturbances.
  • ROVA improves robustness under diverse real-world video perturbations while also improving performance on clean video–question pairs.

A Limitation

The composite reward is effective, but its binary components, fixed equal weighting, external LLM judge, and holistic output-level evaluation leave several design choices for further investigation.

  • Binary format and accuracy rewards provide no partial credit for nearly correct answers or partially structured outputs.The paper suggests softer continuous rewards could provide richer gradients for GRPO optimization.
  • Equal weighting of format compliance, answer correctness, and cross-branch alignment may not suit every perturbation type or reasoning complexity.ROVA does not adapt these weights during training.
  • The alignment reward depends on an external LLM judge whose capabilities and biases can bound the reward signal.Open-source Qwen3-13B alternatives produce comparable results, but domain-specific reasoning remains dependent on judge understanding.
  • Holistic output-level reward evaluation limits assessment to the final answer and overall reasoning output.

B Full Details of Dataset Construction

PVRBench combines UrbanVideo-Bench and VSI-Bench to evaluate video reasoning across outdoor navigation and indoor spatial reasoning under realistic, coherent perturbations.

  • Dataset Integration: PVRBench integrates complete video corpora and question-answer annotations from UrbanVideo-Bench and VSI-Bench.The unified framework spans outdoor urban navigation and indoor spatial reasoning scenarios.
  • UrbanVideo-Bench: UrbanVideo-Bench evaluates aerial Video-LLMs navigating open-ended three-dimensional urban environments.
  • UrbanVideo-Bench: UrbanVideo-Bench contributes 1,547 clips from real-world drone footage, EmbodiedCity Simulator, and AerialVLN Simulator.The sources cover real and simulated urban environments with different resolutions and embodied-navigation settings.
  • Task Coverage: The benchmark’s complementary question-answer distributions cover navigation/action and spatial perception capabilities.UrbanVideo emphasizes navigation/action, whereas VSI-Bench emphasizes spatial perception.
  • VSI-Bench: VSI-Bench evaluates indoor spatial reasoning from egocentric videos and includes fundamental 3D spatial cognition tasks.Its videos come from ARKitScenes, ScanNet, and 3RScan, spanning 288 videos across six indoor environment types and 11 spatial reasoning tasks.
  • Perturbation System: ROVA’s perturbation system generates semantically coherent, temporally consistent, physically plausible corruptions that preserve question answerability.Four modular disturbance categories—lighting, camera, occlusion, and weather—can be applied independently or in combination.
  • Reward and Evaluation: ROVA evaluates clean and perturbed branches using answer matching and reasoning alignment rewards.Answer consistency is binary, while reasoning consistency uses scores of 0, 0.5, or 1.0; both are assessed with GPT-4o.
  • Difficulty-Aware Training: Self-reflective difficulty assessment retains informative samples, filters high-confidence easy samples, and stores difficult samples for later re-evaluation.The assessment uses an LLM judge restricted to the masked video and assigns YES/NO answerability judgments with confidence scores.

D.0.1 Hyperparameter Sensitivity Analysis

ROVA’s sensitivity analysis identifies alignment weights, confidence threshold, group size, and perturbation intensity settings associated with downstream performance and training stability.

  • Alignment Weights: αr = 0.3 and αa = 0.7 improve downstream accuracy while preserving reasoning quality.This setting prioritizes answer alignment over reasoning alignment.
  • Confidence Threshold: τ = 0.8 balances easy-sample filtering against retaining useful training signals.Lower thresholds retain excessive easy samples, whereas higher thresholds discard valuable signals.
  • Difficulty-Aware Training: ROVA uses confidence-aware pruning to skip high-confidence easy samples while retaining or buffering other difficulty classes.The training algorithm stores hard samples in memory and periodically re-evaluates them as the model changes.
  • Group Size: G = 8 provides sufficient group size for stable advantage estimation, with larger groups showing diminishing returns.
  • Perturbation Intensity: η = 0.7 achieves an appropriate perturbation intensity in the sensitivity analysis.

E Additional Experimental Results

Additional experiments show stable reward learning, consistent cross-benchmark and per-task gains, improved performance with more frames, and reduced effective training cost from difficulty-aware selection.

  • Per-Task Performance: ROVA improves per-task accuracy across high-level planning and associative reasoning categories relative to baselines.Radar-chart comparisons cover QwenVL-2.5-7B, Embodied-R-7B, InternVL2.5-8B, Qwen2.5-VL-72B, and Qwen3-VL-13B.
  • Frame Count: Increasing input frames from 16 to 64 improves both baseline and ROVA performance across perturbation types.
  • Reward Dynamics: The total reward converges stably while accuracy rises rapidly, reasoning grows gradually, and temporal reward acts as a temporal regularizer.
  • Cross-Benchmark Evaluation: +14.6% average accuracy on VisBench and +12.9% on UrbanVideo demonstrate consistent cross-benchmark improvements.
  • Sample Stability: Easy-sample retention reaches 97.1% and confidence reaches 0.89 by step 300, or epoch 3.Both metrics increase steadily during training.
  • Self-Reflective Evaluation: Difficult samples have the highest retention rate, identifying them as persistent learning bottlenecks requiring sustained attention.Easy samples show lower and more variable retention.
  • Qualitative Analysis: Qualitative examples show perturbation-aware reasoning under dense fog and heavy snowstorm conditions.Models use visibility limits, depth-distortion cues, multi-frame edge tracking, and snow trajectories in their reasoning.
  • Computational Cost: Difficulty-aware curriculum training reduces effective training cost compared with a naïve dual-branch baseline trained uniformly on all samples.

G.1 Per-Step Cost Decomposition

ROVA adds self-assessment and selective alignment to dual-branch training, reducing downstream computation while maintaining or improving accuracy and reasoning quality. Its measured implementation achieves modest speedups because filtering limits expensive reward and backward stages.

  • ROVA cost decomposition: 0.55–0.65 is the stabilized effective training ratio, so alignment reward calls and backward computation operate on roughly selected samples rather than the full batch.Dual rollouts still run over all N samples for self-assessment before filtering.
  • ROVA cost decomposition: ∼0.4× a standard rollout forward pass is the cost of self-reflective judgment, and downstream savings from filtering exceed this overhead.The judge reuses loaded model weights and evaluates one truncated prompt per sample.
  • Amortized savings: 5.0% cheaper per step is the theoretical ROVA advantage over naïve dual-branch training, despite self-assessment overhead.Savings arise from avoiding alignment API calls and reducing gradient computation for uninformative samples.
  • Measured efficiency: 1.06× wall-clock speedup and +2.3% higher accuracy are achieved by full ROVA relative to naïve dual-branch training.ROVA requires 403s per step versus 428s for the naïve dual-branch baseline.
  • Ablation: 1.9% lower accuracy results when self-assessment is removed, because all samples proceed to expensive alignment and backward stages.Without difficulty-aware filtering, the per-step cost rises to 422s, only 6s below naïve dual-branch training.
  • Cost profile: 70.8% of total time is spent on dual-branch rollout, while self-assessment and memory re-evaluation account for 4.6% and 1.5%, respectively.Alignment reward API calls account for 9.5% of time and benefit from asynchronous batching.

H.4 Comparison with Alternative Reward Designs

ROVA’s default alignment reward outperforms conditional and step-level alternatives on PVRBench. The alternatives degrade performance because their correctness filtering, heuristic segmentation, and added stochasticity produce weaker or noisier training signals.

  • Overall comparison: Neither alternative reward nor their combination improves upon the default ROVA reward; all produce consistent degradation across metrics.The comparison uses Qwen2.5-VL-7B on PVRBench under identical training configurations.
  • Overall comparison: −0.02 in Avg. Acc. and −0.08 in Avg.† are the largest reported drops for the step-level variant.Combining both alternatives does not recover the lost performance.
  • Failure mechanisms: Below 20% of samples remain applicable by mid-training for conditional alignment, and genuinely difficult samples can yield no corrective signal when all G=12 rollouts are incorrect.This limits the usefulness of correctness-conditioned alignment during later training.
  • Failure mechanisms: Heuristic segmentation and frozen-encoder lexical similarity can mis-score semantically equivalent reasoning traces, especially when observation and inference are interleaved.The step-level design decomposes traces into visual observation, spatial/temporal reasoning, and action decision stages.
  • Failure mechanisms: Additional stochasticity increases reward variance, producing noisier GRPO advantage estimates and destabilizing policy updates.The cited sources identify Y+ sampling, edit distance, and heuristic segmentation boundaries as sources of stochasticity.

I Theoretical Analysis

The theoretical analysis models clean and perturbed outputs on a statistical manifold and treats their KL divergence as a local semantic energy. Medium-difficulty sampling is argued to keep updates in a stable trust region with positive dissipation toward an invariant state.

  • Assumptions: DKL(π(z) ∥ π(zϕ)) ≤ ε defines the local-proximity assumption required for learning dynamics to remain within a locally learnable trust region.The assumption applies during stable training steps and requires sufficiently close clean and perturbed output distributions.
  • Statistical-manifold formulation: KL divergence is locally equivalent to the Fisher information metric in a normal neighborhood of M.This local equivalence motivates approximating KL divergence through the manifold’s Fisher geometry.
  • Statistical-manifold formulation: π(z) maps each model representation z to its induced output distribution on the statistical manifold M.Semantic discrepancy between clean and perturbed representations is measured through their induced distributions.
  • Medium-first sampling: Medium-difficulty sampling keeps updates away from singular hard regions and flat easy regions, maintaining ⟨∇θUt, ∇θL⟩ > 0.The analysis associates this positive inner product with gradient alignment and non-vanishing dissipation.
  • Theoretical consequence: The resulting alignment is described as forcing Ut along a dissipative path toward the invariant state.This is the stated theoretical consequence of the trust-region update condition.
Loading 2603.10652v3…