PaperlayerView exact revision on arXiv

Source-linked AI summary

Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMs

Yunheng Li, Guohong Mu, Hao Li, Shengsheng Qian, Dingwen Zhang, Qibin Hou, Ming-Ming Cheng

arXiv:2608.20492v1cs.CV

TL;DR

Video MLLM post-training remains sample-inefficient because on-policy rollouts often lack high-quality positive targets, even with costly chain-of-thought generation. OraRL serializes annotations as oracle rollouts while decoupling them from the policy baseline, and Video-ORA delivers scalable gains across tasks, model sizes, and data budgets with improved efficiency.

  • Problem

    Existing video RL methods sample on-policy groups with few high-quality rollouts, while annotations serve mainly as scoring references rather than reliable positive optimization targets.

  • Method

    OraRL adds each serialized annotation as a detached oracle rollout, estimates policy advantages from on-policy rewards alone, and modulates oracle guidance using the oracle-policy gap.

  • Results

    Video-ORA achieves leading results across seven task families, improves over its backbone from 0.8B to 9B and over GRPO through 100k prompts, while matched evaluations report gains over SFT and GRPO.

  • Takeaways & Limitations

    Annotation-as-rollout is presented as an efficient and scalable reinforcement learning principle for unified video perception.

  • Takeaways & Limitations

    The formulation assumes annotations can be serialized as valid oracle rollouts and evaluated by scalar task rewards; ambiguous, partial, noisy supervision, and learned oracles remain unevaluated.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) have become a prevailing paradigm for unified video perception. However, post-training on large multi-task datasets remains challenging, as existing reinforcement learning methods sample on-policy groups with few high-quality rollouts even with costly chain-of-thought (CoT) generation. In this paper, we study the sample efficiency and scalability of RL post-training for video MLLMs and introduce OraRL. We identify an overlooked role for annotations: Beyond scoring rollouts, each can enter its on-policy group as an oracle rollout, a direct positive optimization target. Direct oracle integration, however, is nontrivial: a high-reward oracle raises the group baseline and inverts otherwise positive policy advantages, a failure we term advantage inversion. At the core of OraRL is a decoupled advantage estimator: policy rollouts determine an oracle-free baseline, while the oracle-policy gap modulates both a directional gain and a separate detached oracle advantage. Sign-balanced pruning improves efficiency: by retaining only the oracle and the strongest rollouts of each sign, OraRL requires just 2.2x the step time of SFT, less than half the 4.9x required by GRPO with CoT. OraRL scales with model size and data, surpassing its backbone from 0.8B to 9B and GRPO up to 100k prompts. Without chain-of-thought, Video-ORA-9B decodes in 130 ms instead of 4,780 ms. Compared with the respective prior best models, it raises temporal mIoU from 62.5 to 66.0, tracking AO from 73.0 to 78.2, segmentation from 64.3 to 70.4, and the three-benchmark spatial-intelligence macro average from 51.0 to 56.1; on VSI-Bench, it scores 73.1 against 55.0 for GPT-5 and 55.1 for Gemini-3-Pro.

1 Introduction

The paper addresses sample-inefficient reinforcement learning for unified video perception by treating annotations as oracle rollouts and introducing OraRL to avoid advantage inversion. Video-ORA achieves broad gains across task families, model scales, data budgets, and efficiency measures.

  • Motivation: Existing video RL methods compare rewards across on-policy rollouts, but rarely obtain precise annotated intervals, boxes, masks, or trajectories as positive anchors.SFT enforces output format but does not distinguish near-correct from clearly incorrect predictions.
  • Method: Annotation-as-rollout serializes each annotation into the model’s response format and appends it as an oracle rollout without requiring chain-of-thought.The mechanism is task-independent and preserves on-policy exploration.
  • Method: Directly mixing a high-reward oracle into the baseline causes advantage inversion, assigning negative advantages to some above-average on-policy rollouts.OraRL instead excludes the oracle from the baseline while retaining it as a detached optimization target.
  • Method: OraRL separates policy advantages from oracle guidance, using the oracle-policy gap for a directional gain and a detached oracle-update weight.This avoids dependence on task-specific reward-shaping semantics.
  • Results: Video-ORA-9B achieves leading results across seven task families, improves over its backbone from 0.8B to 9B, and outperforms GRPO through 100k prompts.Without CoT, it reports best mIoU on all three TimeLens benchmarks, best GOT-10k AO at 78.2, and a VSI-Bench average of 73.1.
  • Efficiency: 1.48× speedup comes from sign-balanced pruning with post-selection moment correction, which retains both advantage signs while reducing update cost.The method computes advantages before pruning and corrects the retained group’s mean and scale.

2 Related Work

Related work spans unified video MLLMs, fine-grained temporal and spatial perception, and reinforcement learning methods such as GRPO. OraRL differs by using paired annotations as oracle rollouts while excluding them from the advantage baseline.

  • Multimodal foundation models for video understanding: Video MLLMs connect visual encoders to language models and represent sampled video frames as visual token sequences within unified architectures.Long-video systems additionally use memory mechanisms and visual-token compression.
  • Fine-grained video perception: Fine-grained video perception includes temporal grounding, highlight detection, reasoning segmentation, and referring video object segmentation, with many MLLM adaptations using supervised fine-tuning.These capabilities remain distinct strands within the broader literature.
  • Reinforcement learning for multimodal models: GRPO estimates advantages by normalizing rewards within on-policy rollout groups without requiring a learned critic, and has been adapted across visual and video reasoning tasks.Existing variants target temporal grounding, referring expression comprehension, segmentation, and tracking.
  • Reinforcement learning for multimodal models: Unlike prior oracle incorporation, OraRL uses paired annotations as teacher-free oracle rollouts but excludes them from the baseline to prevent useful on-policy advantages from inversion.The oracle remains an optimization target while on-policy comparisons are preserved.

3 Methodology

OraRL augments each on-policy group with an annotation-derived oracle rollout while keeping the oracle out of the policy baseline. It separates policy-relative advantages from oracle guidance, then prunes and corrects the retained rollouts before the policy update.

  • Annotation-as-rollout construction: OraRL appends a serialized annotation as an oracle rollout to the on-policy group, preserving all on-policy rollouts and policy-relative exploration.The oracle may encode task-specific outputs such as answer choices, temporal intervals, spatial boxes, trajectories, or segmentation prompts.
  • Advantage inversion: Mixed-group normalization can assign negative advantages to on-policy rollouts that outperform the on-policy mean when the oracle reward is higher.The inversion band has width (rgt−µop)/(n+1), and the oracle also reduces normalized on-policy advantage magnitudes by increasing group dispersion.
  • Decoupled advantage estimation: OraRL computes the baseline from on-policy rewards alone, ensuring that rollouts above the on-policy mean do not receive negative base advantages.The oracle-policy gap is handled separately through directional scaling and a detached oracle advantage.
  • Oracle-gap directional gain: The directional gain amplifies only above-mean rollouts, with clipping preserving the base scale and preventing excessive amplification.The asymmetric scaling is recentered to remove its mean shift while preserving increased separation around the on-policy mean.
  • Detached oracle advantage: The detached oracle advantage measures residual annotation supervision, becoming zero as the on-policy mean reaches the oracle reward and approaching one as the gap grows.Its scale is capped relative to the strongest useful positive on-policy signal, while a 0.05 cap provides a small bootstrap signal when no positive rollout exists.
  • Sign-balanced advantage pruning: Sign-balanced pruning retains the oracle and the strongest positive and negative rollouts, followed by moment correction that restores zero mean, nonnegative oracle advantage, and controlled update scale.Quota allocation is balanced across advantage signs when possible, with unused positions reassigned if one sign lacks candidates.

4 Experiments

Video-ORA-9B achieves leading results across temporal, spatial, tracking, segmentation, video-question-answering, and spatial-intelligence evaluations. OraRL also improves consistently with model size, data scale, backbone choice, and answer-only training, while addressing oracle-injection failures.

  • Main Results: Video-ORA-9B leads all three TimeLens benchmarks, all eight RefCOCO comprehension splits, and all four STVG metrics.It also ranks first on GOT-10k tracking and leads major segmentation results.
  • Main Results: 29.2 J&F points on MeViS and 42.2 on ReasonVOS are Video-ORA-9B’s gains over the Qwen3.5-9B backbone.The backbone cannot produce usable mask prompts independently, while the 4B and 9B models nearly tie on ReasonVOS.
  • Main Results: 73.1 on VSI-Bench is the best reported overall, while Video-ORA-9B also ranks first among open-source models on five of seven Video QA benchmarks.Its spatial-intelligence strengths coexist with weaker planning and rotation performance.
  • Scaling: 66.2 macro average at 9B follows a monotonic rise from 51.8 at 0.8B, with every scale improving all seven task families over its backbone.Tracking’s improvement margin widens from 11.3 points at 0.8B to 32.2 at 9B.
  • Scaling: OraRL gains 5.2 and 3.6 points on video-perception and spatial-intelligence aggregates, versus 2.8 and 3.1 for GRPO across matched data budgets.The comparison spans checkpoints from 6.4k to 100k processed prompts.
  • Ablations: Naive oracle injection lowers the GRPO average from 60.3 to 55.4 and inverts 22.4% of rollouts assigned positive advantages by GRPO.OraRL’s ablations show that directional gain, detached oracle advantage, reward-gap weighting, sign-balanced pruning, and moment correction each contribute to performance or efficiency.

5 Conclusions

OraRL is an efficient and scalable reinforcement learning framework for unified video MLLMs, using annotations as oracle rollouts and improving performance across model scales and data budgets. Its current scope assumes serializable annotations and scalar task rewards, with complex spatial reasoning still lagging some proprietary models.

  • OraRL serializes each annotation as an oracle rollout, providing a reliable positive target while preserving on-policy exploration.
  • Sign-balanced pruning reduces step time from 92.5 to 62.4 seconds with an average loss of 0.4 points.
  • OraRL outperforms SFT and GRPO across controlled tasks and improves consistently across 0.8B–9B models, two backbone families, and data budgets up to 100k prompts.
  • The formulation assumes annotations can be serialized as valid oracle rollouts and evaluated by scalar task rewards.
  • Experiments cover seven task families and two backbone families, while complex spatial reasoning remains below some proprietary models.

A Training Data Construction

The training mixtures normalize multiple source datasets, remove unusable or duplicate examples, and sample SFT and RL data with different task distributions. RL candidates are selected using SFT-checkpoint difficulty estimates, while the configurations are documented in Table 17.

  • SFT and RL cover seven task families but use different sampling distributions, with 284,779 SFT prompts and 100,032 RL prompts.
  • The SFT mixture contains 18% structured and 82% answer-only prompts, whereas RL contains 63% structured and 37% answer-only prompts.
  • Source datasets are normalized to a common schema, and examples with inaccessible media or unrecoverable annotations are discarded.
  • Duplicates are identified using task type, media, question, and answer, with at most two prompts retained per video.
  • The SFT and RL configurations are reported in Table 17.
  • RL subsets emphasize intermediate-difficulty candidates estimated by scoring SFT-checkpoint predictions against annotations with task-specific metrics.

B Training Details

Training initializes models from pretrained backbones, uses shared task-batched rollout groups without chain-of-thought, and applies sign-balanced pruning to reduce optimization cost. Visual processing and hardware settings vary by task and experiment scale.

  • Before RL, language-model parameters are initialized by interpolation between pretrained and SFT parameters, while pretrained visual parameters are retained.
  • For each prompt, the policy generates eight on-policy rollouts and appends the annotation as a ninth oracle rollout, without chain-of-thought generation.
  • At pruning ratio κ = 0.5, sign-balanced pruning retains the oracle, one positive rollout, and two negative on-policy rollouts.
  • The default video pipeline samples at 2 fps, retains at most 128 frames, and applies an 8.4-million-pixel per-clip budget.
  • Controlled 4B comparisons use four nodes with eight NVIDIA H20 GPUs each, while the full 9B run uses eight nodes with full parameter sharding.

C Task Oracles and Rewards

Task adapters serialize annotations into response-format oracle rollouts and score candidate outputs with task-aligned rewards plus format validation. The reward suite covers temporal, spatial, tracking, segmentation, video-question-answering, and numerical spatial-intelligence tasks.

  • Each annotation y is serialized as an oracle rollout, while candidate rollout o receives the task-aligned score R_k(o, y) multiplied by a format-validity indicator.
  • Valid responses require exactly one <answer> block, exclude duplicate blocks, leaked turns, and <think> tags, and may use task-specific spatial JSON.
  • Temporal grounding: Temporal grounding scores predicted and annotated intervals with temporal IoU, and the oracle contains the annotated interval in model response format.
  • Spatial grounding: Spatial grounding uses box IoU, while spatial-temporal grounding combines temporal IoU, strict spatial IoU, temporal coverage, and framewise box quality.
  • Visual tracking: Tracking uses evaluation-aligned average overlap as the task reward, while the oracle contains the complete annotated box trajectory.
  • Mask-aware video segmentation: Segmentation rewards timestamp, box, positive-point, and negative-point predictions, with score caps for absent positive mask points or mask-box IoU below 0.1.

D Full Segmentation Results

The complete segmentation results extend the main-text comparison across every RefCOCO split and report both R@0.5 and cIoU for image benchmarks.

  • The table covers every RefCOCO split and reports both R@0.5 and cIoU.

E Full Spatial-Intelligence Results

The complete spatial-intelligence results report task-level accuracies for MMSI-Bench and MindCube-Tiny, including MMSI-Bench camera, object, and region categories.

  • The table reports accuracies for MMSI-Bench and MindCube-Tiny.MMSI-Bench categories are abbreviated as camera, object, and region.

F Evaluation on ReVSI

Evaluation on ReVSI tests whether Video-ORA’s VSI-Bench gains persist under corrected annotations and matched frame settings. The gains survive overall, but the model shows a substantial robustness drop and a pronounced forward–backward direction asymmetry.

  • Evaluation protocol: ReVSI corrects noisy 3D-derived answers and frame-budget mismatches in VSI-Bench using expert re-annotation and budget-specific answer sets.The evaluation pairs 64-frame inputs with the corresponding answer set and 128-frame inputs with the all-frame answer set.
  • Overall results: Video-ORA’s matched training gain survives ReVSI corrections, improving over the Qwen3.5-9B backbone by 2.5 to 7.6 points on six of seven tasks.
  • Overall results: Video-ORA drops 14.9 points from VSI-Bench to ReVSI, versus 6.3 points for the backbone, so its leading score does not imply greater correction robustness.
  • Relative-direction analysis: At 128 frames, Video-ORA reaches 91.5% on forward questions but 8.3% on backward questions, while Gemini-3-Pro is nearly symmetric at 55.7% versus 56.4%.Matched inputs raise Video-ORA’s forward accuracy over Qwen3.5-9B from 77.4% to 85.8% but lower backward accuracy from 18.4% to 9.1%.
  • Relative-direction analysis: The opposing forward and backward changes leave Video-ORA’s aggregate score nearly unchanged, consistent with VSI-590K’s forward-only templates.

Spatial Grounding STRUCTURED PERCEPTION · 3 EXAMPLES

The qualitative examples span spatial grounding, mask-aware segmentation, tracking, spatial-temporal grounding, video question answering, and spatial intelligence. They illustrate close agreement on several tasks while exposing a temporal-grounding failure mode.

  • Spatial Grounding: Spatial grounding examples show predicted and annotated boxes with IoU of at least 0.98 across three referring expressions.
  • Mask-aware segmentation: Mask-aware segmentation compares predicted boxes and signed point prompts with the annotated mask’s tight box, agreeing within one percent.The model predicts hints for a segmenter rather than emitting a mask directly.
  • Visual tracking: Tracking examples cover changing, nearly stationary, and drifting targets, with mean IoU of at least 0.96 across the full trajectories.
  • Spatial-temporal grounding: Spatial-temporal grounding compares annotated and predicted intervals plus boxes at four seconds, with temporal IoU above 0.87.A wildlife example shows failure when the model keeps one box throughout instead of tracking the subject.
  • Video question answering: Video question-answering examples span attribute recognition, briefly visible details, and counts aggregated over a ten-minute broadcast.
  • Spatial intelligence: Spatial-intelligence examples test first-appearance order, metric distance in metres, and relative direction across unseen egocentric rooms.
Loading 2608.20492v1…