PaperlayerView exact revision on arXiv

Source-linked AI summary

Omni-Streaming Thinking

Enjun Du, Siyi Liu, Ziyu Zheng, Jingyu Li, Yiwen Guo, Yongqi Zhang, Difan Zou

arXiv:2609.15128v1cs.LG

TL;DR

Streaming omni-modal models can commit early visual interpretations before audio evidence matures, allowing contradictions to persist in memory. OST represents such interpretations as future-testable claims, verifies them with retained modality-specific evidence, retracts dependent reasoning, and gates answers until critical reviews pass. It outperforms open baselines across five benchmarks and substantially improves diagnostic discrimination of audio evidence.

  • Problem

    Streaming models must answer from partial synchronized audio–visual observations, but modality-asymmetric evidence maturation can make early interpretations persist as working facts after later audio contradicts them.

  • Method

    OST generates structured evidence, forecasts, and pending claims linked to future modality-specific verification, then retracts dependent states and gates answers using the corrected state.

  • Results

    OST achieves the best open result on five streaming and audio–visual benchmarks, while OST-DiagBench reports d′ = 2.95 versus at most 1.38 for open baselines.

  • Takeaways & Limitations

    Future tests, separate modality retention, dependency-aware retraction, and state rewriting let new evidence revise the reasoning that supports both answer content and timing.

Abstract

from arXiv · show

Streaming omni-modal models must decide what and when to answer from the video chunks and synchronized audio observed so far. Visual cues often support an interpretation before an utterance or sound event is complete. If that interpretation enters memory as a fact, later reasoning can keep relaying it even after audio contradicts it. We call this failure premature cross-modal commitment. We propose Omni-Streaming Thinking (OST), which generates structured outputs that include evidence observed so far, forecasts of future evidence, and claims based on this evidence. Each claim is initially marked as pending and linked to a future verification interval. Audio and visual evidence are stored separately, and OST checks a claim against the evidence from the specified modality at the end of the verification interval. When contradictory evidence is detected, a refutation process reduces the influence of the claim and its dependent states, and then guides a state update using the new evidence. An answer gate decides whether the answer-critical claims meet the conditions for giving a response. Using a frozen Qwen3-Omni-30B-A3B-Instruct backbone with lightweight adaptation, OST outperforms the strongest open baselines on five streaming and audio-visual benchmarks by more than 10% relative on average. We also introduce OST-DiagBench, which holds video fixed and edits audio to test agreement, absence, contradiction, coexistence, and subtitle-speech conflict. OST reaches d-prime = 2.95, compared with at most 1.38 for open baselines, while reducing vision-induced auditory hallucinations.

1 Introduction

Streaming audio–visual models face modality-asymmetric evidence maturation: visual cues may arrive before audio is interpretable, allowing early guesses to persist as facts. OST addresses this by linking unresolved claims to future verification and correcting dependent reasoning before answering.

  • Premature cross-modal commitment occurs when an interpretation becomes a working fact before the evidence needed to test it arrives.An early visual gate estimate can therefore persist through later reasoning even after audio contradicts it.
  • OST records unresolved interpretations as claims specifying expected evidence, verifying modality, due interval, and dependent states.Separate audio and visual retention keeps the relevant evidence available for later checking.
  • When verification finds contradiction, OST attenuates the claim and dependent states, re-decodes the current state, and lets the answer gate wait for scheduled reviews.This connects future testing to the memory and answer decisions that depend on the unresolved interpretation.
  • OST-DiagBench holds video fixed while editing audio to test agreement, absence, contradiction, coexistence, and subtitle–speech conflict.Audio-only comparisons expose cases where models recognize auditory facts in isolation but favor conflicting visual cues when both are present.
  • OST achieves the best open result on five streaming and audio–visual benchmarks and reaches d′ = 2.95 versus at most 1.38 for open baselines.The method is designed to improve both streaming and audio–visual reasoning under cross-modal conflict.

2 Related Work

Prior audio–visual work combines modalities and studies streaming memory, temporal organization, and answer timing. OST is positioned against these lines by explicitly linking unresolved interpretations to later modality-specific verification and memory revision.

  • 2.3 Positioning OST: OST differs by linking unresolved interpretations to claims that later evidence can verify, then using verdicts to update memory and subsequent predictions.Its claim–verify–retract mechanism targets persistent errors from early cross-modal interpretations.
  • 2.1 Audio-Visual Understanding: Audio–visual models have progressed from connecting frozen encoders to language models toward unified processing of text, image, audio, and video.Benchmarks evaluate audio–visual reasoning, including cases where modalities disagree.
  • 2.2 Streaming Models: Streaming methods process observations causally using compact histories, temporal memory organization, token pruning, and retrieval of relevant history.Related systems also distinguish perception, decision, and reaction to study when models should answer.

3 Methods

OST organizes streaming reasoning around structured Omni-States, future-testable claims, modality-specific evidence retention, verification, dependency-aware retraction, rewriting, and gated answering. These components let later evidence revise both the model’s state and the timing of its response.

  • 3.1 Overview of Omni-Streaming Thinking: Each Omni-State contains visual_evidence, audio_state, audio_evidence, conflict, forecast, and sufficiency fields built from retained observations and prior reasoning.The audio_state distinguishes present, absent, and uncertain audio; sufficiency ultimately records WAIT or ANSWER.
  • 3.3 Verification and Causal Retraction: OST updates memory when a claim’s evidence window closes by verifying the relevant modality, retracting contradictions, rewriting the state, and generating new forecasts.The verifier can return CONFIRMED, REFUTED, or UNRESOLVED, while contradiction margins control retraction and rewriting strength.
  • 3.2 Future Claims and Evidence Retention: Each claim stores forecast text, verifying modality, future evidence interval, support scope, dependencies, and an initial reliability score of ρi = 1.Later states and claims retain identifiers for forecasts they use, enabling refutation to reach dependent reasoning.
  • 3.2 Future Claims and Evidence Retention: Separate audio and visual retention budgets preserve pending evidence windows while modality-specific eviction manages the bounded memory.Evidence windows remain pinned until their claims settle or expire, preventing dense visual tokens from displacing verification audio.
  • 3.3 Verification and Causal Retraction: Refutation propagates through an acyclic provenance graph by lowering effective reliability for a claim and its dependent spans, summaries, and states.Reliability scores only decrease, so retraction persists when reasoning is reused or compressed.
  • 3.4 Answer Gating: The answer gate answers only when evidence is sufficient and answer-supporting claims no longer await scheduled review, otherwise waiting until termination with low confidence when unresolved dependencies remain.This gate jointly controls answer content and response timing.

4 Experiments

OST is evaluated on streaming accuracy, auditory-evidence use under cross-modal agreement and conflict, and the contribution of its verification and correction stages. It achieves strong benchmark performance while reducing visual bias and gains depend on retaining, verifying, and propagating evidence.

  • 4.1 Main Results on Streaming and Audio-Visual Benchmarks (RQ1): 87.8 average accuracy exceeds StreamOV’s 81.6 with audio–visual context, while OST achieves the highest online accuracy across SOVBench-O question types.OST uses Qwen3-Omni-30B-A3B-Instruct across the evaluated streaming and audio–visual benchmarks.
  • 4.2 Cross-Modal Bias Diagnosis on OST-DiagBench (RQ2): Baseline models can recover audio-only information yet favor conflicting visual cues, with Qwen3-Omni’s SWAP attribution falling to 35.29 from 79.80 audio-only donor recall.With stale subtitles visible, naive-streaming CLASH scores fall to 0.59, 4.90, and 0.00 despite audio-only scores of 95.69, 80.78, and 94.12.
  • 4.2 Cross-Modal Bias Diagnosis on OST-DiagBench (RQ2): d′ = 2.95 combines 95.49 CLEAN source recall with a 10.39 MUTE hallucination rate, versus at most 1.38 for open baselines.OST also leads on SWAP attribution and MIX joint recall, and reaches 46.47 CLASH accuracy versus at most 4.90 for open baselines.
  • 4.3 Ablation Studies (RQ3): 82.5-to-74.2 SOVBench-O Recall and 53.1-to-40.7 Video-Holmes TCI drops show that retaining and organizing evidence support longer dependencies.These are component ablations of OST with other evaluation settings held fixed.
  • 4.3 Ablation Studies (RQ3): 87.8-to-83.6 and 2.95-to-1.95 drops show that delayed modality-specific verification and typed ownership matter to OST’s performance.Removing claim propagation, verdict-conditioned rewriting, decoding contrast, or hard support checks also reduces downstream metrics.

5 Conclusion

OST treats early interpretations as future-testable claims whose verification can revise dependent reasoning before answering. This enables new audio evidence to update both answer content and timing while improving performance across streaming and audio–visual benchmarks.

  • OST links early interpretations to future verification and dependent states, allowing contradictory audio to revise the reasoning used for answers.Typed verification, dependency-aware retraction, and state rewriting update the state used to answer.

A Implementation Specification

The implementation represents streaming evidence, claims, verdicts, and memory in structured Omni-States with bounded retention and dependency-aware verification. Claims remain reviewable through scheduled intervals, while refutation reliability propagates through their ancestry.

  • Retention and state representation: OST stores question-conditioned evidence, Omni-States, claims, and verdicts in an active reasoning ledger alongside modality-specific retained observations.The context combines observed video and audio with earlier states and claim records.
  • Retention and state representation: The memory pyramid compresses 4 s, 16 s, and 64 s summaries into a long-term root while preserving claim ancestors and updated reliability scores.Capacity constraints keep unsettled claims at the finest level until settlement or expiry.
  • Claim registration and verification: Each registered claim specifies a verifying modality, future interval, support scope, dependencies, and lifecycle metadata before entering the registry.Admission requires a query-critical proposition and a strictly future verification interval.
  • Claim registration and verification: The verifier confirms, refutes, or leaves claims unresolved using modality-specific evidence, with retry and expiry rules preserving uncertainty when evidence remains insufficient.Unresolved claims may be retried with right context, then expire with a low-confidence answer flag.
  • Refutation and dependency propagation: Refutation caps a claim’s reliability and propagates the least reliable ancestor’s influence to descendants without multiplying depth-dependent decay.This provenance rule prevents downstream states from becoming more reliable than a refuted ancestor.

B.3 CLASH: Subtitle–Speech Fact Conflict

CLASH and conditioned audio interventions test whether models follow speech and sound changes when visual content remains fixed. The paired evaluations expose errors caused by adopting conflicting visible facts or visually suggested sounds.

  • CLASH and controlled interventions: 510 reviewed CLASH items pair identical edited audio with and without conflicting subtitles to test subtitle–speech fact conflicts.The dataset uses aligned fact replacements, audibility checks, visible text, and neutral questions.
  • CLASH and controlled interventions: Adding video changes 485, 388, and 480 previously correct audio-only answers into errors for Qwen2.5-Omni-7B, MiniCPM-o 2.6, and Qwen3-Omni-30B-A3B-Instruct.482, 376, and 478 of these changes respectively switch to the visible old fact.
  • CLASH and controlled interventions: The intervention suite varies agreement and contradiction through CLEAN, MUTE, SWAP, and MIX while holding video fixed and controlling donor properties.MUTE and SWAP remove the source; MIX retains both source and donor.
  • Failure-mechanism panels: Conditioned panels vary semantic distance, onset, evidence strength, and acoustic robustness to isolate mechanisms behind cross-modal failures.The panels compare NEAR-SWAP with FAR-SWAP and test compression, reverberation, band limitation, and additive noise.
  • Failure-mechanism panels: In Figure 7, MUTE elicits a visually suggested guitar report, MIX can omit the donor, and donor loudness or onset changes whether source, joint, or donor recognition is reported.With neutral video, the preserved MIX waveform changes the report toward both guitar and dog; late donor onset is recognized more reliably than early or middle onset.

B.7 Judge and Metrics

The evaluation separates audio, subtitle, source, donor, and joint recognition so that correct reporting depends on the intervention condition. Metrics pair recall with hallucination or omission measures rather than treating all answers as equivalent.

  • Judging and event metrics: The judge matches answers against source, donor, background, and visual-only references, while CLASH separately identifies audio, subtitle, both, or neither facts.Only an audio-only report is correct for CLASH-Acc.
  • Judging and event metrics: MIX uses source, donor, and joint recall because both events are present, whereas MUTE and SWAP measure reports of the removed source as visual hallucinations.Recall and hallucination rates are reported together, and source-only MIX answers count as donor omissions.

B.8 Evaluation and Statistical Protocol

The evaluation compares completed-clip offline runs with causal streaming runs over nominal 1 s chunks. Statistical comparisons are paired by source or item, with uncertainty estimated using clustered bootstrap procedures.

  • Offline evaluation uses completed clips, whereas naive streaming carries history across nominal 1 s chunks and asks at chunk end.Short container tails are merged into the preceding chunk.
  • OST consumes the stream causally and answers when its gate fires.
  • Comparisons use source-cluster bootstrap intervals, exact McNemar tests for paired binary outcomes, and item-level clustering for CLASH.Repeated MIX donors use a two-way source/donor bootstrap; uncertainty describes the selected challenge set.

B.9 Factorial-Core Results

Table 8 evaluates all models on 510 matched groups using a shared prompt and event-level judge. The factorial-core results separately report VHR for MUTE and SWAP and multiple recall measures for MIX.

  • 510 matched groups are evaluated for all models with PROMPT_V2 and the event-level judge.
  • Table 8 reports factorial-core results from 510 matched groups drawn from 1,068 admitted sources.
  • VHR is reported separately for MUTE and SWAP, while MIX reports source, donor, and joint recall.

C.1 Implementation and Evaluation Protocols

Matched offline and streaming runs use the same prompt and decoding settings, with experiments run on an eight-GPU server. Answer normalization maps generated responses to benchmark endpoints through task-specific procedures.

  • Matched offline and streaming runs use identical prompt and decoding settings on a server with eight NVIDIA GPUs.The server provides approximately 1.13 TB of aggregate GPU memory.
  • A shared normalizer maps free-form outputs to benchmark endpoints using lexical or embedding matching, constrained selection, or benchmark scoring.Multiple-choice, binary, and open-ended answers use different normalization paths.
  • Options are supplied during normalization after streaming reasoning, while CLASH uses a deterministic alias matcher.

C.2 Full Results on Standard Benchmarks

The benchmark results are organized across SOVBench, StreamingBench, Video-Holmes, Daily-Omni, and OmniVideoBench. Tables provide overall or task-specific results, with baselines reproduced from cited sources.

  • SOVBench results appear in Table 1, while Tables 9–13 provide category and duration breakdowns for the other four benchmarks.Baseline values are reproduced from the sources cited in each caption.
  • StreamingBench reports separate audio–visual and visual-only task breakdowns in Tables 9 and 10.
  • Table 9 reports StreamingBench audio–visual QA, with StreamOV baseline average 73.1 and Qwen2.5-Omni-7B average 49.6 under the stated averaging rule.
  • Table 10 reports StreamingBench visual-only QA percentages using StreamOV baselines.
  • Table 11 reports Video-Holmes accuracy at 32 frames, with micro-accuracy averaged over 1,837 questions.
  • Daily-Omni accuracy is reported at 1 FPS, while OmniVideoBench reports accuracy by audio type and duration.
Loading 2609.15128v1…