PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniReasoning: Pushing the Limits of Audio-Visual Joint Reasoning

Junming Lin, Yuxuan Wang, Zhenxin Lei, Yuxin Liu, Ruixun Liu, Yinsong Yan, Ling Wang, Minghao Han, Yunfei Chu, Shun Lei, Xueyao Zhang, Qize Yang, Jin Xu, Yiwu Zhong

arXiv:2609.39490v1cs.CVcs.AI

TL;DR

Audio-visual joint reasoning remains poorly evaluated and insufficiently elicited when benchmarks, data, and learning methods treat modalities independently. OmniReasoning combines a benchmark, evidence-grounded data engine, and modality-factored self-distillation; its model improves on targeted and broader video evaluations.

  • Problem

    Existing benchmarks, training data, and learning methods often treat modalities independently, leaving the reliability of audio-visual joint reasoning unclear and some benchmark questions answerable from one modality.

    Source: §1 · p. 2

  • Method

    OmniReasoning pairs a benchmark and OmniQA’s timestamped, modality-specific evidence chains with MFSD, which uses clue contexts to assign token-level credit for modality contributions and interactions.

    Source: §1 · pp. 2–3

  • Results

    50.0% accuracy on OmniVideoBench and 42.5% on OmniReasoningBench improve over Qwen3-Omni-30B-A3B-Thinking by 12.8 and 9.3 percentage points, respectively.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Results suggest that constructing, annotating, and optimizing for cross-modal evidence dependencies can strengthen audio-visual joint reasoning and broader video understanding.

    Source: §1 · p. 3

    MFSD’s interpretation of its contrast as a token-wise increment in conditional PMI depends on the joint-consistency assumption; positive interaction alone does not establish a positive joint-clue gain over no clues.

    Source: §3.3 · p. 7

Abstract

from arXiv · show

Recent advances have enabled unified omni-modal models in understanding audio, vision, and language. However, existing benchmarks, training data, and learning methods largely treat the modalities independently, leaving the capability of audio-visual joint reasoning poorly evaluated and insufficiently elicited. We address this gap with a benchmark, data engine, and learning method. First, we introduce OmniReasoningBench, a benchmark where both audio and visual evidence are indispensable. It comprises 1,150 multiple-choice and open-ended questions across two tasks, reasoning over video and reasoning beyond video. Second, we develop a data engine OmniQA. It automatically constructs evidence-grounded QA pairs that explicitly necessitate audio-visual joint reasoning, together with time-stamped clue chains that guide the annotation of thinking process. Besides our benchmark, this engine produces training data OmniReasoning-SFT-112K and OmniReasoning-RL-19K. Finally, we propose an on-policy self-distillation method Modality-Factored Self-Distillation (MFSD). It evaluates each sampled response under modality-specific clue contexts, disentangling the contributions of individual clues and their cross-modal interactions for token-level credit assignment. With our training data and learning method, our model OmniReasoning-30B-A3B achieves 50.0% on OmniVideoBench and 42.5% on OmniReasoningBench, improving the base model Qwen3-Omni-30B-A3B-Thinking by 12.8 and 9.3 percentage points, respectively. Moreover, it delivers substantial gains on general and long-video benchmarks, including Video-MME-v2. We hope our work offers a solid step for facilitating future research in omni-modal joint reasoning.

1 INTRODUCTION

OmniReasoning addresses uncertainty about audio-visual joint reasoning with a benchmark, evidence-grounded training data, and MFSD. The resulting model improves on targeted joint-reasoning and broader video benchmarks.

1 INTRODUCTION

  • Whether omni-modal models can reliably reason across heard and seen evidence remains unclear, and prior benchmark ablations do not ensure that both modalities are necessary.Visual-only evaluations of several benchmarks retain substantial accuracy, limiting their ability to distinguish joint reasoning from single-modality understanding.Source: §1 · p. 2
  • Existing benchmarks can leave substantial questions answerable from one modality, while OmniReasoningBench constructs 1,150 questions requiring complementary audio and visual evidence across two reasoning settings.The settings are reasoning over video and applying video-derived information to a new scenario.Source: §1 · p. 2
  • OmniQA constructs evidence-grounded audio-visual questions with timestamped clues and dependency chains, which support OmniReasoning-SFT-112K and structured supervision in OmniReasoning-RL-19K.Verified reasoning chains synthesize thinking processes for supervised fine-tuning.Source: §1 · pp. 2–3
  • MFSD evaluates sampled responses under no-clue, audio, visual, and joint clue contexts to assign token-level credit for individual modalities and their interaction.It combines the interaction contrast with support from joint clues to encourage reasoning based on complementary evidence.Source: §1 · p. 3
  • 50.0% accuracy on OmniVideoBench and 42.5% on OmniReasoningBench improve over Qwen3-Omni-30B-A3B-Thinking by 12.8 and 9.3 percentage points, respectively.The model also improves on general and long-video benchmarks, including Video-MME-v2.Source: §1 · p. 3

2 RELATED WORK

Prior work spans omni-modal models and benchmarks, audio-visual instruction data, and reinforcement-learning approaches to reasoning. OmniReasoning distinguishes its benchmark through necessary cross-modal evidence and its data and learning method through explicit modality-specific clues.

2 RELATED WORK

  • Prior benchmarks cover omni-modal understanding, cross-modal dependence, clue-grounded counting, and knowledge acquisition from instructional video; OmniReasoningBench requires joint reasoning over audio and visual observations.The benchmark also includes reasoning beyond video, applying video-derived knowledge to new inputs.Source: §2 · p. 3
  • OmniVideo-100K uses entity-anchored scripts and clue-guided questions, while OmniVideo-R1 filters questions for cross-modal grounding; OmniQA adds explicit evidence, dependency chains, and separate audio and visual clues for RL scoring.OmniQA also extends video-derived knowledge to new inputs through reasoning beyond video.Source: §2 · p. 3
  • Figure 2 distinguishes reasoning over video, which connects observations across events, from reasoning beyond video, which applies video-derived knowledge to a new scenario.Orange and blue mark audio and visual clues, and numbered markers link evidence to timestamps.Source: Figure 2 · p. 4
  • Where prior RL and self-distillation methods include group-relative rewards, sequence weighting, key-token attribution, or token-level feedback, MFSD combines a centered interaction contrast from separate and joint clues with RLSD weighting.MFSD retains RLSD’s outcome verifier and policy objective.Source: §2 · p. 4

3 BENCHMARK, DATA ENGINE, AND LEARNING METHOD

OmniReasoning links its benchmark, training data, and learning method through explicit dependencies between audio and visual evidence.

3.1 OMNIREASONINGBENCH: A BENCHMARK FOR AUDIO-VISUAL JOINT REASONING

  • 750 questions test reasoning over video and 400 test transfer beyond video; joint inputs beat a single-modality oracle by 9.9–26.7 points on over-video questions.The benchmark includes multiple-choice and open-ended formats; every over-video answer requires chaining audio and visual evidence.Source: §3.1 · pp. 4–5

3.2 OMNIQA: A DATA ENGINE FOR AUDIO-VISUAL JOINT REASONING

  • OmniQA builds timestamped audio and visual clues, verified QA pairs, and evidence chains that guide thinking generation and provide RL context.The released corpora contain 112,463 SFT samples with synthesized thinking and 18,991 human-validated RL questions with evidence annotations.Source: §3.2 · pp. 5–6, Figure 3 · p. 5

3.3 MODALITY-FACTORED SELF-DISTILLATION: A LEARNING METHOD

  • MFSD scores each sampled response under no-clue, audio, visual, and joint contexts, using joint support and non-additive interaction for bounded token-level credit.Interaction is positive when the combined clues raise a token’s likelihood by more than the sum of their separate effects; centered interaction redistributes credit within a response.Source: §3.3 · pp. 6–8

4 EXPERIMENTS

OmniReasoning-30B-A3B improves on its base model on OmniReasoningBench and public video benchmarks; under matched RL conditions, MFSD outperforms GRPO and RLSD on the reported measures.

4 EXPERIMENTS

  • The evaluation withholds reference evidence chains and requires direct answers to open-ended questions.The model starts from Qwen3-Omni-30B-A3B-Thinking; RL methods share a 150-step budget and a global batch of 256 generated responses per update.Source: §4 · p. 8

4.1 AUDIO-VISUAL JOINT REASONING AND GENERALIZATION

  • 42.5% on OmniReasoningBench and 50.0% on OmniVideoBench exceed the base model by 9.3 and 12.8 points, respectively.Gains span both OmniReasoningBench settings and question formats, and every audio-type and duration group on OmniVideoBench.Source: §4.1 · p. 9, Table 3 · p. 9

4.2 CONTRIBUTION OF MFSD

  • MFSD improves OmniReasoningBench by 6.4 points over SFT, versus 2.2 for GRPO and 3.1 for RLSD, and beats SFT by 0.7–4.5 points on existing benchmarks.GRPO, RLSD, and MFSD use the same SFT checkpoint and RL budget.Source: §4.2 · pp. 9–10, Table 4 · p. 10

5 CONCLUSION

OmniReasoning combines a benchmark, evidence-grounded training data, and modality-aware reinforcement learning to improve audio-visual joint reasoning. The paper also documents its generative-AI use and provides training and evaluation details for reproducibility.

5 CONCLUSION

  • OmniReasoning combines a benchmark, timestamped-clue data from OmniQA, and MFSD to improve audio-visual joint reasoning and general and long-video understanding.The resulting OmniReasoning-30B-A3B substantially improves on existing and proposed benchmarks.Source: §5 · p. 10
  • Generative AI assisted with OmniQA dataset construction and writing, while authors reviewed the work and verified technical content and synthetic data.The authors state that generative AI was not used for core scientific contributions, methodology, experiments, analysis, or scientific claims.Source: AI USE STATEMENT · p. 11
  • Training details, model configurations, optimization objectives, hyperparameters, implementation settings, and evaluation protocols are documented in the paper and appendices for reproducibility.Source: REPRODUCIBILITY STATEMENT · p. 11

A IMPLEMENTATION AND EVALUATION DETAILS

This appendix specifies the SFT and reinforcement-learning configurations, evaluation sampling settings, and policy objective implementation used in the experiments.

A IMPLEMENTATION AND EVALUATION DETAILS

  • SFT uses 112,463 examples, a 7 × 10−6 learning rate, global batch size 128, and frozen visual and audio encoders.Videos are sampled at 2 fps, with at most 256 frames and 200,704 pixels per frame.Source: A IMPLEMENTATION AND EVALUATION DETAILS · p. 13
  • GRPO, RLSD, and MFSD runs use 150 steps, 64 GPUs, global batch size 256, eight rollout trajectories per question, and a 10−6 learning rate.MFSD also uses text-clue factorization, actor weights for each scoring view, sequence-wise centering, and EMA coefficient 0.9 for κ, capped at 10.Source: A IMPLEMENTATION AND EVALUATION DETAILS · p. 13
  • Evaluation on OmniReasoningBench, OmniVideoBench, LVOmniBench, Video-MME-v2, and Video-MMMU samples video at 2 fps, up to 512 frames, without audio truncation.Source: A IMPLEMENTATION AND EVALUATION DETAILS · p. 13

A.1 POLICY OBJECTIVE

  • The policy objective’s masked response-token reduction and KL term anchor the actor to the SFT reference policy, while teacher scores and advantage weights are detached.MFSD adds three scoring passes, reuses media features, and requires neither additional rollouts nor teacher parameters.Source: §A.1 · p. 13

A.2 MFSD IMPLEMENTATION AND COMPONENT ANALYSIS

MFSD calibrates and combines joint-clue and interaction signals for token weighting, with centering and clue-availability rules shaping the resulting credit assignment.

A.2 MFSD IMPLEMENTATION AND COMPONENT ANALYSIS

  • MFSD calibrates joint-clue gain and centered interaction scales using valid response-token batch standard deviations, excluding padding and non-response positions.The calibration approximately aligns the signal scales; clipping and smoothing do not guarantee exact variance matching.Source: §A.2 · p. 13
  • Centering makes the interaction term redistribute credit among response tokens without changing their total token-score sum, while removing response-wide additive offsets.Source: §A.2 · p. 13
  • When either clue set is absent, MFSD sets β_i to zero and uses joint-clue-only weighting; joint-clue support can remain useful even when it comes from one modality.Source: §A.2 · p. 13
  • β balances joint support against calibrated interaction, while λ controls how strongly token weights modify the outcome advantage; both warm up to 0.5.Setting β to zero removes interaction, and setting λ to zero restores unweighted outcome advantage.Source: §A.2 · p. 13
  • MFSD reuses the actor, media, response, and prefixes across four scoring views; detached evidence scores and weights keep clue-conditioned derivatives out of the policy gradient.The three clue-conditioned views reuse rollout and media features, without extra generation or separate teacher parameters.Source: §A.2 · p. 14

A.3 BENCHMARK EVALUATION PROTOCOLS

The evaluation protocol defines the benchmark composition and test inputs, and compares joint audio-visual evidence with either modality alone across evidence-chain lengths.

A.3 BENCHMARK EVALUATION PROTOCOLS

  • OmniReasoningBench contains 750 reasoning-over-video and 400 reasoning-beyond-video questions, with multiple-choice and open-numeric formats across both tasks.Reasoning beyond video includes 288 video-plus-figure and 112 video-plus-text inputs.Source: §A.3 · p. 14
  • At evaluation, models receive the source video, question, and any additional figure, while reference evidence chains are withheld.For reasoning-beyond-video ablations, the additional figure or table stays fixed while only the source video's audio and visual streams vary.Source: §A.3 · p. 14
  • Audio-visual joint inputs outperform either modality alone for all three advanced Omni-LLMs in every evidence-chain-length group.Source: Figure 6 · p. 14

A.4 MODALITY ABLATIONS

Joint audio-visual inputs outperform single-modality alternatives, including an oracle combining their separate successes. The appendix also reports OmniVideoBench breakdowns and formal properties of MFSD’s interaction scoring, centering, and bounded reweighting.

A.4 MODALITY ABLATIONS

  • Across three Omni-LLMs, reasoning-over-video accuracy falls from 42.3–50.1% with joint inputs to 10.9–14.9% with video alone and 16.3–20.4% with audio alone.The comparison supports the importance of joint evidence.Source: §A.4 · p. 15
  • On prior benchmarks, joint inputs add 4.9–15.8 points over visual-only inputs, while 44.9–68.8% of questions remain answerable without audio; OmniReasoning questions show larger visual-only drops.For OmniReasoning multiple-choice and open-ended questions, visual-only accuracy is 17.2% and 8.7%, versus 42.7% and 41.9% with joint inputs.Source: §A.4 · p. 15, Table 5 · p. 15
  • Joint inputs exceed the single-modality oracle by 9.9–26.7 percentage points, so their gains cannot be explained by combining audio-only and video-only successes.For Qwen3.5-Omni-Plus, 212 questions were answered correctly only with joint inputs, while 77 were answered by the oracle but missed jointly.Source: §A.4 · p. 15, Table 6 · p. 15

A.5 OMNIVIDEOBENCH BREAKDOWN

  • OmniReasoning-30B-A3B improves over its base model in every OmniVideoBench audio-type and video-duration group.Source: §A.5 · p. 15

B MATHEMATICAL PROPERTIES OF MFSD

  • MFSD’s information-theoretic analysis varies privileged audio, visual, and joint clues while retaining the original audio-video input and sampled response prefix.Source: B MATHEMATICAL PROPERTIES OF MFSD · p. 16

B.1 BAYESIAN INTERPRETATION OF THE INTERACTION SCORE

  • Under joint consistency, the interaction score measures joint likelihood support beyond individual likelihood ratios and increments conditional PMI, not correctness or positive joint support.Positive interaction and positive joint support are distinct; the joint-clue gain and interaction remain separate branches.Source: §B.1 · p. 16

B.2 SEQUENCE CENTERING

  • Sequence centering removes response-wide additive offsets while preserving relative active-token ordering and making the interaction contribution zero-mean in the log-credit signal.The zero-sum property applies to the linear score, not the later nonlinear exponentiation, clipping, and interpolation.Source: §B.2 · p. 17

B.3 BOUNDED REWEIGHTING AND GRADIENT STRUCTURE

  • MFSD’s bounded reweighting preserves the sign of the original advantage and limits its change to a λϵw fraction of its magnitude.Privileged clues rescale student-token gradients without reversing them or introducing derivatives through clue-conditioned scores.Source: §B.3 · p. 17

C REWARD AND EVALUATION PROTOCOLS

RL rewards are binary over the human-validated OmniReasoning-RL-19K questions, with different verification procedures for multiple-choice and open-numeric answers. The qualitative comparison presents paired model responses on identical questions across five benchmarks.

C REWARD AND EVALUATION PROTOCOLS

  • Binary RL rewards cover 18,991 human-validated questions, using exact option-letter matching for multiple choice and a DeepSeek-V4-Flash judge for open-numeric answers.The same judge and prompt are used for RL rewards and all open-ended evaluation.Source: C REWARD AND EVALUATION PROTOCOLS · p. 18

D QUALITATIVE COMPARISON OF MODEL RESPONSES

  • The qualitative comparison pairs complete responses to identical questions across five benchmarks, including their full thinking text and final answers.Cases cover temporal alignment, speaker and object reference, motion comparison, and applying video-derived knowledge to new inputs.Source: D QUALITATIVE COMPARISON OF MODEL… · p. 19

D.1 OMNIVIDEOBENCH: DIFFERENT DESCRIPTIONS OF THE SCENE AND INSCRIPTIONS

The examples compare base and post-trained responses to cross-modal video questions, including how OmniReasoning combines spoken cues and visual evidence for spatial, color, motion, and speaker–object descriptions. They show where responses match reference answers and where models differ in what direction or object the evidence supports, without independently verifying the video evidence.

D.1 OMNIVIDEOBENCH: DIFFERENT DESCRIPTIONS OF THE SCENE AND INSCRIPTIONS

  • OmniReasoning’s answer selected United States, matching the reference, while the base response selected Apollo and United States.The comparison describes the responses as reporting different inscriptions; it does not independently verify the video timestamps or inscriptions.Source: §D.1 · pp. 19–20

D.2 VIDEO-MMMU: DIFFERENT STATED TEMPERATURES AND ANSWER CHOICES

  • OmniReasoning selected 0.545 or 54.5% using a 10°C cold reservoir, matching the reference; the base response assumed 20°C and selected 0.527 or 52.7%.The passage says the comparison does not independently verify which temperature or timestamp appears in the video.Source: §D.2 · pp. 20–21

D.3 VIDEO-MMMU: DIFFERENT CLAIMED DIAGRAM CORRESPONDENCES

  • OmniReasoning selected diagram a, matching the reference answer A, while the base response selected diagram i; the question’s wording contains an empty referent after “in.”The diagram correspondences are claims in the responses and are not independently verified in the comparison.Source: §D.3 · p. 22

D.4 OMNIVIDEOBENCH: ALIGNING ORCHESTRAL SOUND WITH PHYSICAL CONTACT

  • For the first physical contact, OmniReasoning associated the event with an orchestral chord and selected A, matching the reference, whereas Qwen3-Omni attributed it to descending strings.The responses differ on the musical change associated with the contact.Source: §D.4 · p. 23

D.5 OMNIVIDEOBENCH: GROUNDING A SPOKEN PHRASE IN THE VISIBLE INGREDIENT

  • OmniReasoning linked “No, nada, aquí andamos” to the bell pepper being handled and selected A, while Qwen3-Omni described an ingredient absent from the options and substituted egg.The reference answer is bell pepper.Source: §D.5 · p. 25

D.6 LVOMNIBENCH: DISTINGUISHING BACKGROUND MUSIC FROM SCENE AUDIO

  • For salon background music, OmniReasoning considered the recorded audio characteristics and selected editing-added music, matching the reference, while Qwen3-Omni relied on an ambient-music expectation.The comparison concerns the models’ reasoning and answers about the music source.Source: §D.6 · p. 26

D.7 LVOMNIBENCH: TRACKING A MUSIC TRANSITION IN A LONG VIDEO

  • OmniReasoning tracked the sequence of music cues and linked the requested fade-out to an outdoor pathway, matching reference answer A.The responses locate different musical transitions.Source: §D.7 · p. 27

D.8 LVOMNIBENCH: RESOLVING SPATIAL REFERENCE FROM SPEECH AND GESTURE

  • The spatial-reference question asks which side of the Christmas tree the cat tree is on based on the blogger’s gesture; the reference answer is the left side.The supplied passages provide the question and reference answer but no model-response comparison.Source: §D.8 · p. 29
  • For the cat-tree question, OmniReasoning binds the spoken reference to a leftward gesture and selects option C, opposite the direction assigned by the other model.The supplied result states that the models assign opposite directions to the referenced gesture.Source: §D.8 · p. 29

D.9 VIDEO-MME-V2: MATCHING A SPOKEN CUE TO AN OBJECT COLOR

  • When asked about the phone on the right during “I would be kicking myself so much,” OmniReasoning selects teal (C), while Qwen3-Omni uses a later model-color association.The supplied question asks for the phone’s color at that spoken cue.Source: §D.9 · p. 30

D.10 VIDEO-MME-V2: COMPARING MOTION AMPLITUDE ACROSS DEMONSTRATIONS

  • OmniReasoning identifies the broom’s swing amplitude as much larger than the hammers’, while Qwen3-Omni selects C after treating the wording as a typo.OmniReasoning selects D, the option describing a very large broom swing relative to the hammers.Source: §D.10 · p. 31

D.11 VIDEO-MME-V2: IDENTIFYING MOTION DIRECTION AT A NARRATION CUE

  • For the duck-motion narration cue, both responses identify a circular trajectory but disagree on direction; OmniReasoning selects counter-clockwise option F.The reference answer is F, “Walking multiple irregular counter-clockwise circular trajectories.”Source: §D.11 · p. 33

D.12 OMNIREASONINGBENCH / Reasoning over video: LINKING SPEAKER IDENTITY TO AN EARLIER OBJECT

  • For the presenter saying “I think the Harden takes the cake for sure,” OmniReasoning links the speaker to his screen position and the shoe shown during his introduction.The supplied example identifies the answer as the presenter on the right holding a green, orange, and black Nike PG 2 shoe.Source: §D.12 · pp. 34–35

D.13 OMNIREASONINGBENCH / Reasoning over video: LOCATING A SCENE BETWEEN TWO SPOKEN ANCHORS

This example tests whether a model can locate a visual scene using two spoken temporal anchors. The scene between “Unicorn” and “wireless technology” is a sailor waving signal flags.

D.13 OMNIREASONINGBENCH / Reasoning over video: LOCATING A SCENE BETWEEN TWO SPOKEN ANCHORS

  • The two spoken references define the interval in which the visual scene must be found.The question uses the narrator’s mention of “Unicorn” and “wireless technology” as its temporal boundaries.Source: §D.13 · p. 36
  • The scene after “Unicorn” and before “wireless technology” is the sailor in a white uniform waving signal flags, option D.The sailor appears at 01:37, between the spoken anchors at 01:33 and 01:52.Source: §D.13 · pp. 36–37
  • The painted bust of Lord Nelson is excluded because it appears before the “Unicorn” reference, at 00:54.The sailor scene, by contrast, appears at 01:37 within the interval.Source: §D.13 · p. 37

D.14 OMNIREASONINGBENCH / Reasoning beyond video: APPLYING A VIDEO-DERIVED RATE TO A NEW SALES TABLE

These examples test reasoning beyond video by applying information from a video to a new table or practical scenario. One requires scaling sales estimates by a video-derived rate; the other transfers a demonstrated woodworking procedure.

D.14 OMNIREASONINGBENCH / Reasoning beyond video: APPLYING A VIDEO-DERIVED RATE TO A NEW SALES TABLE

  • Qwen3-Omni compares the table’s Canada sales estimates directly and selects Product Y, while OmniReasoning applies the video’s rate and selects Product X.The direct-estimate response treats $150,000 for Product Y as highest; applying the 30% rate to Product X’s US sales gives $300,000.Source: §D.14 · pp. 37–38
  • Applying the competitors’ stated 30% business share in the Canadian market to US sales, Product X has the highest potential Canadian sales at $300,000 (option F).Source: §D.14 · pp. 37–38

D.15 OMNIREASONINGBENCH / Reasoning beyond video: TRANSFERRING A DEMONSTRATED WOODWORKING PROCEDURE

  • The woodworking procedure transfers a demonstrated sequence: plane, rub with a small flat board, plane again, rub again, then check with a try square.OmniReasoning includes the rubbing step, unlike Qwen3-Omni’s conventional planing-and-checking sequence.Source: §D.15 · pp. 39–40

E BENCHMARK TASK ATLAS

The atlas presents one source-recorded case for each of ten reasoning-over-video and nine reasoning-beyond-video task types. The selected examples illustrate task design, not the correctness of every annotation or model explanation.

Reasoning over video

  • Background-and-music and temporal reasoning use audio cues to locate or order visual events, such as identifying an action during sung lyrics or ordering spoken and visual events.The atlas’s examples include a lyric-anchored action question and a question ordering four events from an introduction and movie segment.Source: Figure 7 · p. 41, E BENCHMARK TASK ATLAS · p. 45
  • Causal and counting questions combine spoken information with visual evidence, asking about the origin of pollution alongside a fish observation or summing repetition targets for selected scenes.The examples ask who was responsible for overfeeding and the frogfish’s color, and total repetitions for exercises in specified settings.Source: E BENCHMARK TASK ATLAS · pp. 41–42
  • Other over-video examples link identity or speech to earlier clothing, exercise demonstrations, a dragon, or a scoreboard at a spoken moment.They cover ego-centric understanding, perception, reference, relationship, and spatial reasoning.Source: E BENCHMARK TASK ATLAS · pp. 42–45

Reasoning beyond video

  • Beyond-video examples apply video-derived knowledge to a blade-sharpening diagram, wound configurations, or a new tournament catch log.These illustrate case study analysis, comparative reasoning, and cross-scenario transfer.Source: E BENCHMARK TASK ATLAS · pp. 46–48
  • Further transfer examples address a de-esser threshold, an acrylic-painting error, profile length, and population-normalized rates using new figures or conditions.These illustrate design and optimization, error critique, multi-step quantitative reasoning, and problem-solving adaptation.Source: E BENCHMARK TASK ATLAS · pp. 49–51

F SUPERVISED TRAINING EXAMPLES

This appendix presents synthetic supervised-training examples from the production task taxonomy, pairing questions and answers with audio-visual evidence chains and complete stored responses; these are training targets, not outputs from the evaluated model. Across nine task types, an example identifies the defensive read prompting a specific hip-rotation move and distinguishes it from reads prompting a step-back.

F.1 A01 / AUDIO SOURCE GROUNDING; F.2 A02 / SOUND EVENT REASONING

  • Audio-source and sound-event examples require comparing modalities: a quoted speaker is off-screen during a graphic, while narration adds “consider the following” beyond the displayed text.The first example identifies the host’s voiceover during the quote; the second finds the extra spoken phrase immediately after the on-screen wording.Source: §F.1 · p. 52, §F.2 · p. 53

F.3 A03 / MUSIC & ATMOSPHERE UNDERSTANDING

  • The music-understanding example matches file-copying visuals with upbeat electronic music heard during the overlapping interval.The file explorer shows copying from 02:00–02:22, overlapping with the music from 02:13–02:22.Source: §F.3 · p. 54

F.4 A04 / CROSS-MODAL REFERENCE RESOLUTION

  • Cross-modal reference resolution links the on-screen Chatham House title to the expert’s spoken claim that the detonation’s reported size was relatively modest.The visual banner names John Nilsson-Wright; the corresponding audio statement compares the detonation with North Korea’s 2013 test.Source: §F.4 · p. 56

F.5 A05 / AUDIO-ANCHORED PERCEPTION

  • Audio-anchored perception uses the phrase “source to sink” and a shirt-color match across scenes to identify the pointed-at machine as an EOS M 400.The maroon shirt links the man in the meeting room to the person pointing at the labeled machine.Source: §F.5 · p. 57

F.6 A06 / COUNTING

  • The counting example uses the narrator’s “Dot time strip technology” cue to identify the goldfish-filter segment, whose feature list contains nine text boxes.The visual list is described as three sets of three boxes.Source: §F.6 · p. 58

F.7 A07 / TEMPORAL REASONING & ORDERING

  • The temporal-ordering example aligns a warrior’s first appearance with his traditional shout, distinguishing it from the horn, later group chant, and poi performance.The described warrior appears at 01:05 with the shout; the horn occurs earlier, while the group chant and poi performance occur later.Source: §F.7 · p. 59

F.8 A08 / SPATIAL REASONING

  • The spatial-reasoning example uses the spoken CPU-fan instruction to anchor a frame in which the blue and pink motherboard ports are right of the header.The source passage notes that the camera angle is reversed from its usual orientation.Source: §F.8 · p. 60

F.9 A09 / CAUSAL REASONING

  • The causal-reasoning example pairs the trainer’s stated defensive cue— the screener’s defender sucking up to the frame—with his subsequent hip-rotation demonstration.The trainer describes looking for that defensive action before pivoting on his inside foot and opening his outside hip.Source: §F.9 · p. 61
  • For the hip-rotation move, the cue is the screener’s defender sucking up to the frame (option D); sagging or corralling instead prompts a step-back.The move involves pivoting on the inside foot and opening the outside hip.Source: §F.9 · pp. 61–62

F.10 A10 / EMOTION & SENTIMENT ANALYSIS

These examples use audio, visual, and combined evidence to answer questions about emotion, actions, events, and relationships. They also include questions requiring synthesis and application of information to a new figure.

F.10 A10 / EMOTION & SENTIMENT ANALYSIS

  • The young man’s emotion shifts from calm to fearful after his mother says his principal called.The evidence chain connects his calm speech, her statement, and his startled, anxious reaction.Source: §F.10 · p. 62

F.11 A11 / HYPOTHETICAL & PREDICTIVE REASONING

  • Without the pre-selection filter for ports and net labels, other objects inside the selection area would also be selected.The example combines the visual use of the filter with the narrator’s explanation of its effect.Source: §F.11 · p. 63

F.12 A12 / CHARACTER RELATIONSHIP & INTERACTION

  • Just before leaving, the vlogger says “Well, take care” and fist-bumps the Don Knotts statue.The example distinguishes this final interaction from earlier dialogue in the scene.Source: §F.12 · p. 64

F.13 A13 / EVENT UNDERSTANDING & STATE CHANGE

  • After the narrator announces a simple query, the user types a new query into the editor.The answer joins the spoken cue to the subsequent on-screen action.Source: §F.13 · p. 65

F.14 A14 / CROSS-SCENE ASSOCIATION & LONG-CONTEXT

  • The later character shown during the discussion of super humans has white feathered angel wings.This example asks the reader to connect an early mention of a tech innovator and a character type with a later depiction; its source entry has no separately recorded structured upstream evidence.Source: §F.14 · pp. 66–67

F.15 A15 / SUMMARIZATION & SYNTHESIS

  • Fast-paced death metal and a dark, surreal album cover combine to create a dark and chaotic mood.The evidence chain identifies the aggressive, chaotic audio and the dark visual setting as the cues for the answer.Source: §F.15 · pp. 67–68

F.16 A16 / EGO-CENTRIC & FIRST-PERSON UNDERSTANDING

  • When highway traffic is first heard passing below, the cyclist points right with their left hand.The example links the first traffic sound to the gesture observed while crossing the bridge.Source: §F.16 · pp. 68–69

F.17 B01 / PROBLEM-SOLVING ADAPTATION

  • The Blog Header is incorrectly configured: the video classifies blog quality as High, but the figure shows Standard.The other listed settings match their intended uses: Mobile Wallpaper uses Standard, Email Newsletter uses High, and Photo Print uses Maximum.Source: §F.17 · pp. 69–70

F.18 B02 / CASE STUDY ANALYSIS

These examples require applying information from videos to figures, real-world constraints, and alternative plans. They cover case-study identification, transfer, quantitative reasoning, comparison, and design choice.

F.19 B03 / CROSS-SCENARIO TRANSFER

  • Only Drone B meets the requirement of at least 26 minutes of actual flight time: its realistic estimate is about 28 minutes.Drone A’s estimate is about 24–25 minutes, below the stated threshold.Source: §F.19 · pp. 71–72

F.20 B04 / MULTI-STEP QUANTITATIVE REASONING

  • Seven complete gift packs can be made, limited by the seven pillow covers available.The example derives 20 cushion covers from four sets and adds five individual pillow covers to the two in the bedsheet set.Source: §F.20 · pp. 73–74

F.21 B05 / COMPARATIVE REASONING

  • Plan A takes 8.33 hours, less than Judi Hu’s original project time of about 9 hours; the other listed plans take longer.The calculation uses 8 minutes per flower and 90 minutes per patch.Source: §F.21 · pp. 74–75

F.22 B06 / DESIGN / OPTIMIZATION

  • Pick A, whose stem is thick and hard to cut, matches the demonstrated bending-and-tying attachment method.The video’s decision rule favors bending for stems too thick to cut easily and cutting thinner stems.Source: §F.22 · pp. 75–76

F.23 B07 / ERROR CRITIQUE

These examples ask readers to critique a recommendation, identify a demonstrated procedure, and transfer a technique to a new context.

F.23 B07 / ERROR CRITIQUE

  • The matte screen-protector recommendation rests on a false premise: the Garmin Nuvi 2555 LMT screen is already anti-glare.The video describes the screen as designed for anti-glare, making an additional matte protector unnecessary and potentially clarity-reducing.Source: §F.23 · pp. 76–77

F.24 B08 / PROCEDURE / PLANNING

  • For the cleaned-out Tashtego 2.0 toiletry roll, the demonstrated next step is placing it in a washing machine.The video shows the bag being put into the machine after it is rinsed.Source: §F.24 · p. 78

F.25 B09 / CROSS-DOMAIN TRANSFER

  • For a forward twirl with a hose nozzle, initiate with the palm up and the thumb diving down.The example transfers the instructor’s bo-staff technique to the firefighter’s scenario.Source: §F.25 · p. 79

Related papers

Loading 2609.39490v1…