PaperlayerView exact revision on arXiv

Source-linked AI summary

OPERA: A Unified Omnimodal Progressive Spatio-Temporal Reasoning Agent for Referring Video Segmentation

Jingchen Ni, Yuji Wang, Shannan Yan, Haoru Li, Sitong Chen, Chun Yuan

arXiv:2609.33338v1cs.CV

TL;DR

Referring video segmentation with heterogeneous queries faces temporal frame-selection and spatial reasoning gaps. OPERA uses a unified, three-stage agent for progressive reasoning across both axes; it reports state-of-the-art results on OmniAVS and Ref-AVS and zero-shot transfer to standard benchmarks.

  • Problem

    Referring video segmentation must map heterogeneous multimodal queries to per-frame masks, while existing methods use naive frame selection and lack a structured chain from semantic understanding through grounding to masks.

    Source: §1 · p. 1, §3.1 · p. 2

  • Method

    OPERA uses one MLLM backbone with temporal coarse-to-fine frame filtering, cross-modal distillation, and GRPO-enhanced grounding followed by dense mask propagation.

    Source: §1 · p. 1, §3.4 · p. 4

  • Results

    OPERA reports state-of-the-art results on OmniAVS and Ref-AVS and zero-shot transfer to standard referring video segmentation benchmarks.

    Source: §5 · p. 7

  • Takeaways & Limitations

    Semantic distillation provides the largest gain, while efficient temporal selection approaches the single-pivot oracle ceiling and structured outputs enable reliable mask propagation.

    Source: §5 · p. 7

    OPERA assumes access to the full video, placing streaming outside its scope.

    Source: §5 · p. 7

Abstract

from arXiv · show

Referring video segmentation with heterogeneous multimodal queries---spanning text, audio, and reference images---demands both robust cross-modal understanding and precise spatio-temporal reasoning. We propose OPERA (Omnimodal Progressive spatio-tEmporal Reasoning Agent), a unified reasoning agent built on a single MLLM that performs dual-axis progressive reasoning via three specialized stages. Along the temporal axis, a Temporal Reasoning Agent narrows the frame search space through coarse-to-fine filtering to identify the most informative key frame. Along the spatial axis, a Distillation Agent establishes what to locate via cross-modal semantic distillation, and a Grounding Agent enhanced with GRPO determines where the target appears, with dense mask propagation completing the pixel-level output. OPERA sets a new state of the art on OmniAVS and Ref-AVS and transfers zero-shot to standard referring video segmentation benchmarks.

1 Introduction

OPERA addresses temporal and spatial reasoning gaps in referring video segmentation with three specialized stages sharing one MLLM backbone. It reports state-of-the-art results on OmniAVS and Ref-AVS and zero-shot transfer to standard benchmarks.

1 Introduction

  • Existing methods select grounding frames with fixed heuristics and can anchor on occluded or ambiguous frames, while also lacking a structured chain from semantic understanding to region grounding and pixel masks.Such frame choices can cause localization errors that propagate throughout the video.Source: §1 · p. 1
  • The Temporal Reasoning Agent filters candidate frames coarse-to-fine, while the Distillation and Grounding Agents establish what to locate and where it appears before dense mask propagation.The Distillation Agent converts non-textual cues into unified textual representations, and the Grounding Agent uses GRPO-enhanced chain-of-thought reasoning.Source: §1 · p. 1
  • OPERA decomposes referring video segmentation into three specialized stages on one MLLM backbone and handles all eight text, speech, sound, and image query combinations without modality-specific heads.Its contributions also report state-of-the-art performance on OmniAVS and Ref-AVS and zero-shot transfer to standard referring video segmentation benchmarks.Source: §1 · pp. 1–2
  • OPERA’s dual-axis design separates temporal frame filtering from spatial reasoning, and controlled ablations attribute its reported gains to the reasoning protocol rather than the backbone.The temporal axis progresses from uniform sampling through vision-language filtering to LLM-based reranking.Source: §1 · p. 2

2 Related Work

Related work spans text-, audio-, and omnimodal referring segmentation, as well as omni-modal foundation models and multimodal reasoning. OPERA’s evidence-axis input rewriting is distinguished from Omni-R1’s resolution- and compute-oriented rewriting.

Omnimodal Referring Segmentation

  • Text-guided referring segmentation spans static-image datasets such as RefCOCO and video benchmarks such as Refer-YouTube-VOS (Seo, Lee, and Han 2020) and MeViS.Ref-AVS (Wang et al. 2024b) uses joint audio–text guidance, while OmniAVS supports eight expression types over text, speech, sound, and image.Source: §2 · p. 2

Omni-Modal Foundation Models

  • ImageBind and LanguageBind (Zhu et al. 2023a) learn shared embeddings, while recent omni-modal models unify text, image, and audio for fusion and reasoning.Omni-R1 uses input rewriting for low-resolution reasoning and high-resolution grounding; OPERA instead translates non-textual cues into source and appearance constraints, making their mechanisms complementary.Source: §2 · p. 2
  • Related efficiency work includes quantization-aware training (Lv et al. 2026) and speculative decoding for diffusion language models (Cui et al. 2026).The cited work also describes reinforcement learning as enhancing multimodal reasoning, including Omni-R1’s RL-based keyframe selection and instruction rewriting.Source: §2 · p. 2

3 Method

OPERA maps heterogeneous video queries to per-frame masks through three specialized agents sharing one MLLM backbone. Its reasoning proceeds across temporal and spatial axes, from selecting a key frame and identifying and grounding the target to propagating masks across the video.

3.1 Problem Formulation and Overview

  • Eight query types pair text or speech with optional sound and image cues, and OPERA maps each video-query pair to a per-frame binary mask sequence.Three specialized stages share one MLLM backbone and pass structured outputs along a deterministic chain.Source: §3.1 · p. 2

3.2 Distillation Agent

  • The Distillation Agent turns non-textual cues into textual hints and rewrites them into a unified grounding prompt with minimal changes to core evidence.It transcribes speech, describes environmental sounds, and captions reference images conditioned on the query text.Source: §3.2 · pp. 2–3

3.3 Temporal Reasoning Agent

  • The Temporal Reasoning Agent selects a key frame by uniformly sampling K frames, retaining top-Nc candidates through vision-language filtering, then semantically reranking them with an LLM.For speech queries, the vision-language filtering uses the transcript produced by the Distillation Agent; an invalid reranking index falls back to the highest-scoring filtered candidate.Source: §3.3 · p. 3

3.4 Grounding Agent

  • The Grounding Agent takes the unified prompt and selected frame to produce pixel-coordinate bounding boxes, using one interface across modality combinations.Its structured output interleaves a reasoning trace with localization; GRPO optimizes candidate outputs using relative rewards, and the reward balances box localization quality with format compliance.Source: §3.4 · p. 4

3.5 Dense Mask Propagation

  • The dense propagation module uses the predicted box to initialize an object representation on the key frame, then propagates masks forward and backward across the video.It emits a mask for every frame.Source: §3.5 · p. 4

4 Experiments

OPERA is evaluated on omnimodal and audio-visual referring video segmentation, zero-shot transfer benchmarks, component ablations, and qualitative examples. Results report strong benchmark performance while probing the contributions and costs of its reasoning stages.

4.1 Implementation Details

  • OPERA uses Qwen2.5-Omni 3B or 7B, CLIP filtering with K=16 and Nc=4, and SAM2; the Grounding Agent trains one epoch on 54,304 OmniAVS samples.Other benchmarks are evaluated zero-shot.Source: §4.1 · pp. 4–5

4.3 Audio-Visual Referring Video Segmentation

  • On Ref-AVS, OPERA-7B reaches J &F of 67.8 on Seen and 74.4 on Unseen, surpassing OISA by 12.6 and 12.7 points, respectively.The Seen gain over Omni-R1-7B is 20.6 points, while Unseen performance is essentially on par.Source: Table 2 · p. 5, §4.3 · p. 5

4.4 Referring and Reasoning Video Segmentation

  • Zero-shot OPERA-7B scores 72.2 on R-DAVIS17, 66.1 on R-YTVOS, and 57.0 on ReVOS, exceeding the strongest in-domain competitor on ReVOS by 9.4 points.Baselines are fine-tuned on their target benchmarks, so these comparisons are not apples-to-apples.Source: Table 3 · p. 5, §4.4 · pp. 5–6

4.2 Omnimodal Referring Video Segmentation

  • OPERA-7B achieves an overall J &F of 53.3 on OmniAVS, surpassing OISA by 12.2 points and Omni-R1-7B by 6.7 points across eight modality splits.The largest improvements are on audio-dominant Splits III, V, and VI: 26.9, 23.8, and 24.9 points.Source: §4.2 · p. 5

4.5 Ablation Study

  • J &F peaks at K=16 and Nc=4; removing speech, image, and sound distillation costs 3.4, 3.5, and 4.5 points on their respective query subsets.Source: Figure 3 · p. 6, §4.5 · pp. 6–7
  • The Distillation Agent yields the largest component gains, 8.5 points on OmniAVS and 9.9 on Ref-AVS; adding Rfmt reaches 53.3 on OmniAVS, and structured outputs provide parseable SAM2 prompts.Swapping to Qwen3-Omni-30B-A3B lifts J &F from 43.1 to 56.2 under the same recipe.Source: §4.5 · pp. 6–7
  • VISA’s 53.4 is indistinguishable from OPERA’s 53.3 but takes 23.19 s; a K=16 oracle reaches only 54.1, leaving 0.8 points of single-pivot headroom.Overall reasoning latency is 7.69 s per video, 2.0 s above Omni-R1-7B for 6.7 J &F points.Source: §4.5 · pp. 6–7, Table 5 · p. 6

4.6 Qualitative Results

  • Qualitative examples show speech negation reasoning and a trimodal case where a reference image and chirping cue jointly isolate a small insect.The figure presents propagated masks, selected key frames, and grounding traces for four queries.Source: §4.6 · p. 7, Figure 4 · p. 7

5 Conclusion

OPERA combines dual-axis progressive reasoning and structured outputs, achieving state-of-the-art results on OmniAVS and Ref-AVS and zero-shot transfer to standard benchmarks. The appendix’s controlled comparison places its temporal policy near the single-pivot ceiling while highlighting full-video availability as a scope boundary.

5 Conclusion

  • OPERA achieves state-of-the-art results on OmniAVS and Ref-AVS and transfers zero-shot to standard referring video segmentation benchmarks.Its dual-axis process narrows the frame search temporally and uses distillation and GRPO-optimized grounding spatially; structured outputs support mask propagation.Source: §5 · p. 7
  • OPERA assumes the full video is available, placing streaming outside its scope; sliding-window selection and multi-pivot grounding are proposed as extensions.The authors also identify stronger omni-modal backbones evaluated under matched latency budgets as a natural extension.Source: §5 · p. 7

Appendix

  • The appendix covers selector comparisons, output-protocol and backbone controls, implementation and reward details, runtime, additional benchmark results, and qualitative examples with failure analysis.The selector comparison uses a controlled pipeline and reports a single-pivot oracle.Source: Appendix · p. 11, A Key Frame Selection: External… · p. 11

A Key Frame Selection: External Selectors and the Single-Pivot Oracle

  • The single-pivot oracle reaches 54.1, only 0.8 points above TRA’s 53.3, while VISA reaches 53.4 but takes 23.19 s versus TRA’s 7.69 s.The comparison holds the Distillation Agent, Grounding Agent, and SAM2 fixed; the authors characterize TRA as efficient rather than as solving key frame selection.Source: A Key Frame Selection: External… · p. 11

B Output Protocol and Backbone Generality

The controls distinguish structured output formatting from free-form reasoning and test whether the GRPO recipe transfers across backbones. Structured think-plus-bbox output outperforms bbox-only output, while a second backbone also improves under the recipe.

B Output Protocol and Backbone Generality

  • 53.3 for structured think-plus-bbox output exceeds 46.7 for bbox-only output and the untrained 37.9 on OmniAVS.The bbox-only control removes the reasoning trace while holding data, optimizer, reward budget, and GRPO steps fixed; the authors attribute the benefit to parseable outputs for SAM2, not free-form reasoning itself.Source: Table 8 · p. 11, B Output Protocol and Backbone… · p. 11
  • J &F rises from 43.1 to 56.2 on Qwen3-Omni-30B-A3B, a 13.1-point gain, compared with 15.4 points on Qwen2.5-Omni-7B.The authors caution that two backbones do not establish generality in full.Source: B Output Protocol and Backbone… · p. 11

C.1 GRPO Training Configuration

OPERA trains its Grounding Agent with full-parameter GRPO, while its MLLM-based stages share one checkpoint at inference. Inference uses greedy decoding, distinct from the sampling used for training rollouts.

C.1 GRPO Training Configuration

  • GRPO training runs for 4,000 steps on OmniAVS with groups of 8 candidate outputs, β = 0.04, clipping ratio ϵ = 0.2, and reward weight α = 0.8.Training uses full-parameter optimization on 8 NVIDIA H20 GPUs, with AdamW at an initial learning rate of 5×10^-7; no held-out validation split is used for model selection.Source: §C.1 · p. 11

C.2 Per-Agent Inference Configuration

  • DA, TRA Level-3, and GA share one Qwen2.5-Omni-7B instance at inference; only GA is updated by GRPO, while DA and TRA Level-3 are frozen and zero-shot.The stages use the same post-GRPO checkpoint without weight copies or model reloads between stages.Source: §C.2 · pp. 11–12, Table 9 · p. 12
  • Inference uses greedy decoding with a 2,048-token maximum, whereas GRPO training rollouts use temperature 0.5 and repetition penalty 1.15.TRA Level-3 makes sequential captioning and reranking calls, each capped at 2,048 tokens.Source: §C.2 · p. 12

C.3 Reward Function Specification

The reward combines localization quality and output format checks, while the appendix reports threshold-dependent training reward behavior. The format checks assess both a complete think-chain structure and valid bounding-box output.

C.3 Reward Function Specification

  • The reward combines localization IoU and format terms as R = α RIoU + (1−α) Rfmt.The format term checks think-chain completeness and bounding-box formatting; each check contributes half of that term.Source: §C.3 · p. 12
  • Invalid bounding-box output receives zero format reward during training; at inference, a failed parse becomes an empty prediction and no mask is propagated.If the TRA Level-3 reranker returns an invalid frame index, the system falls back to the highest-scoring Level-2 CLIP candidate.Source: §C.3 · p. 12
  • At δ=0.5, reward rises steadily to 0.80; δ=0.3 reaches ≈0.75 but decouples from localization precision, while δ=0.7 stays near 0.20 for 1,500 steps.The curves compare δ values over 4,000 training steps; solid lines show raw reward and dashed lines show EMA.Source: Figure 5 · p. 12

C.4 Reward Ablation Studies

Reward ablations show that localization-threshold and reward-weight choices strongly affect performance, while the prompt templates convert non-textual cues, candidate frames, and rewritten queries into inputs for the three agents.

C.4 Reward Ablation Studies

  • δ=0.5 yields the best J &F of 53.3; δ=0.3 reaches only 26.7 despite reward convergence near 0.75, while δ=0.7 yields 21.0 after sparse rewards.Across the three settings, J &F spans 32.3 points; at δ=0.7 the reward stays near 0.20 for the first 1,500 steps.Source: §C.4 · p. 12
  • At α=0.8, J &F reaches 53.3, versus 50.3 with pure IoU supervision, 37.4 at α=0.6, and 25.0 at α=0.4.The authors attribute the drop below α=0.8 to weakened localization supervision and inaccurate predictions that satisfy the format criterion.Source: §C.4 · pp. 12–13
  • The δ and α ablations identify both values as consequential GRPO hyperparameters that the authors report explicitly rather than leaving at defaults.The threshold controls the IoU criterion; α balances localization and format rewards.Source: §C.4 · pp. 12–13

C.5 Prompt Templates

  • Each non-textual modality receives a dedicated Distillation Agent prompt that produces a compact textual hint, merged into rewritten query q′.The Temporal Reasoning Agent's Level-3 captioning prompt describes each candidate frame, then a reranking prompt selects the keyframe; the Grounding Agent uses q′ to reason and predict boxes in absolute pixel coordinates.Source: §C.5 · p. 13

D Runtime and Computational Cost

Runtime measurements show how OPERA's components trade latency for J &F, with Distillation Agent gains making the full pipeline outperform Omni-R1-7B. The reported prompt setup uses modality-specific hints and a caption-then-rerank keyframe selection cascade.

D Runtime and Computational Cost

  • At 7,693 ms per video, full OPERA reaches 53.3 J &F versus Omni-R1-7B's 46.6, with 2,028 ms additional latency.Measurements cover 6,791 OmniAVS test samples on one NVIDIA H20; SAM2 propagation is excluded from all timings.Source: D Runtime and Computational Cost · p. 13
  • GRPO training alone adds 4.2 J &F, from 37.9 to 42.1, at 229 ms latency cost, or 54.5 ms per point.The passage identifies this as the lowest overhead of any pipeline component.Source: D Runtime and Computational Cost · p. 13
  • DA adds 8.5 J &F for a net 443 ms pipeline increase, while TRA without DA adds 4,004 ms for only 2.7 points.DA itself costs 1,022 ms and reduces GA latency by 675 ms; its net efficiency is 52 ms per point.Source: D Runtime and Computational Cost · p. 13
  • DA prompts turn each non-textual modality into a textual hint merged into q′, while TRA Level-3 captions candidate frames and reranks them to select a keyframe.The TRA captioning prompt omits its “Previous frame” line for k=0; an invalid reranking output falls back to the highest-scoring Level-2 CLIP candidate.Source: Figure 6 · p. 13, Figure 7 · p. 14
  • Activating DA raises TRA latency only 96 ms, from 3,877 to 3,973 ms, a 2.5% change.The authors interpret this near-invariance as showing that TRA cost is driven by video-processing volume rather than query complexity.Source: D Runtime and Computational Cost · pp. 13–14

E Additional Quantitative Results

OPERA improves over Omni-R1-7B on ReVOS and MeVIS, with gains varying across ReVOS task splits and model scales. The smaller model remains competitive on ReVOS but falls below larger competitors on MeVIS.

E Additional Quantitative Results

F Extended Qualitative Results and Failure Analysis

The qualitative examples trace OPERA from text-only and speech queries through multimodal combinations, while the failure analysis shows how an early-stage error can persist through the deterministic pipeline.

F Extended Qualitative Results and Failure Analysis

  • Six examples cover five of eight OmniAVS query types, ordered from text-only and speech to dual-modal and trimodal inputs.Their visual annotations distinguish DA hints, TRA candidate frames and reranking, and GA reasoning and predicted boxes.Source: F Extended Qualitative Results and… · p. 15
  • None of the six examples selects the clip's first frame; the passage links this pattern to the 3.0-point first-frame gap reported in Tab. 5.The Level-2 shortlists span the video, and Level-3 selects indices such as 12, 26, and 33.Source: F Extended Qualitative Results and… · p. 15
  • Failure modes include ambiguous speech or multi-reading reference images, cluttered CLIP candidates, same-class instance confusion, and SAM2 drift after long target absence.A reference image can bind to the wrong same-class instrument; committing to one key frame then lets SAM2 propagate that identity through the video.Source: F Extended Qualitative Results and… · p. 15
  • For text-plus-image queries, DA distills the reference image into a compact visual description; for speech-plus-sound, it merges speech and sound hints into one grounding query before keyframe selection.Other examples show image distillation identifying a person across instances and joint sound-and-image hints informing the grounding query.Source: Figure 10 · p. 16, Figure 9 · p. 16, Figure 11 · p. 17

Related papers

Loading 2609.33338v1…