PaperlayerView exact revision on arXiv

Source-linked AI summary

TRACE: Temporal Audit and Condition-aware Evaluation of Streaming Video Understanding

Yibo Ma, Qianqian Zhang, Peng Liu, Tiancheng Zhao

arXiv:2609.30670v1cs.CLcs.CV

TL;DR

Streaming-video evaluations may leave evidence validity, history processing, and response triggers unspecified, although scores depend on these conditions. TRACE combines temporal audits, a controlled Core–Adapter protocol, and operational reporting; its evaluations show that similar task scores can accompany substantially different operational outcomes.

  • Problem

    Streaming-video evaluations often report task scores without specifying when evidence is valid, how visual history is maintained, or how responses are triggered.

    Source: §1 · p. 1

  • Method

    TRACE pairs temporally audited QA evidence and proactive response windows with a controlled Core–Adapter protocol and multidimensional operational reporting.

    Source: §1 · p. 1, §7 · p. 12

  • Results

    Similar task scores can accompany substantially different completion, answer validity, workload, response timing, false alarms, and missed target windows.

    Source: §1 · pp. 1–2, §7 · p. 12

  • Takeaways & Limitations

    Streaming-video results are interpretable when temporal validity, execution conditions, and operational behavior are made explicit alongside task accuracy.

    Source: §7 · p. 12

    Findings cover tested systems on visual-only, single-instruction tasks at 1 FPS, and proactive tasks emphasize positive triggers.

    Source: §8 · p. 12

Abstract

from arXiv · show

Streaming video understanding requires models to interpret evidence as it arrives, yet current evaluations often report task scores without specifying when evidence becomes valid, how visual history is maintained, or how responses are triggered. As a result, similar scores may correspond to different workloads, failure modes, and operational behavior. We introduce TRACE (Temporal Audit and Condition-aware Evaluation), a condition-aware benchmark and evaluation framework that makes these factors explicit. TRACE combines temporally audited visual tasks with evidence timing and instruction-dependent trigger annotations, a unified causal Core--Adapter protocol that controls information availability while recording actual history processing and response events, and multidimensional reporting of answer quality, timeliness, response-selection behavior, workload, completion, and reliability. On 1,240 records from 517 videos, we evaluate eight publicly available models or systems in eight configurations. We find that nearly identical QA accuracy can mask substantial differences in completion, answer validity, and generation workload, while proactive performance separates into response quality, response delay, false alarms (responses emitted while no target window is currently valid and a later one remains), and missed target windows. These results show that streaming-video performance should be interpreted as execution-conditioned system behavior rather than a single score. Our benchmark and code can be accessed at \href{https://github.com/om-ai-lab/trace-bench}{https://github.com/om-ai-lab/trace-bench}.

1. Introduction

Streaming-video evaluations must account for when evidence is valid, how visual history is processed, and when responses are triggered. TRACE makes these conditions explicit and shows that similar task scores can accompany different operational outcomes.

1. Introduction

  • Unlike offline long-video evaluation, streaming systems must process observations incrementally and respond as questions arrive or monitored conditions become true.Comparisons with offline video models require accounting for the evidence available when each response was produced [10].Source: §1 · p. 1
  • A streaming score depends on evidence validity, history maintenance, response triggering, and the processing and failures that occur before a response.Systems with similar task scores may differ in state maintenance, history replay, timing, redundant output, workload, completion, and answer validity.Source: §1 · p. 1
  • TRACE combines temporally audited visual tasks, a causal Core–Adapter protocol, and multidimensional reporting of quality, timing, output, workload, completion, and reliability.The protocol controls available video while recording history processing and response events; task annotations specify valid evidence and instruction-dependent proactive triggers.Source: §1 · pp. 1–2
  • Nearly identical QA accuracy can coincide with differences in completion, output volume, and invalid-output rates, while similar proactive scores can conceal differences in timing, false alarms, and missed windows.Differences in history processing and evaluation boundaries also affect what a reported score represents.Source: §1 · pp. 1–2

2. Related Work

Prior work evaluates long-context understanding, causal online video access, and proactive response timing, but these controls do not by themselves make streaming evaluations directly comparable. TRACE situates its evaluation around explicit temporal validity, execution conditions, and operational behavior.

Long-video understanding

  • Long-video benchmarks test event recognition, information aggregation, and temporal reasoning, typically with the complete video or a presampled context available before inference [5] [25] [31] [17] [21].They do not directly establish whether a model can process continuously arriving evidence or respond when an interaction requires it.Source: §2 · p. 2

Streaming and online video evaluation

  • Streaming benchmarks constrain causal access and organize temporal relationships, but causal visibility alone does not specify how systems form visual state or what processing precedes a response [12] [18] [22] [6].These works include StreamingBench [12], RIVER [18], S-EMBER [22], and OVBench [6].Source: §2 · p. 2

Proactive and interactive evaluation

  • Proactive benchmarks assess waiting for evidence or interactive response behavior [9] [23] [29] [24] [16], while TRACE focuses on visual-only, single-instruction evaluation that jointly audits timing, execution, and operations.The cited works include OVO-Bench [9], ProactiveVideoQA [23], OmniPro [29], OmniMMI [24], and OmniInteract [16].Source: §2 · p. 2

Positioning of TRACE

  • Table 1 distinguishes evidence timing and trigger annotations from metrics including QA accuracy, latency, parsability, proactive accuracy, response timing, false-alarm/miss diagnostics, workload, and completion.Its purpose is to locate TRACE within the released evaluation landscape, not rank benchmarks; redundant output is reported separately because its semantics vary across protocols.Source: §2 · p. 2
  • Causal access and response timing do not ensure direct comparability when evidence boundaries, history mechanisms, response triggers, or timing boundaries differ.TRACE makes these conditions explicit and evaluates them alongside task quality and operational behavior.Source: §2 · p. 2

3. TRACE Evaluation Framework

TRACE treats streaming-video scores as conditional on evidence validity, execution conditions, and observed operational behavior. It links temporal annotations to a causal Core–Adapter protocol and reports quality, timing, response behavior, workload, completion, and reliability.

3.1. Design Principles and Overview

  • TRACE links temporally audited tasks, a causal Evaluation Core, model-specific Adapters, and task-specific scoring to measure behavior under declared execution conditions.The task set supplies reviewed QA evidence times and proactive trigger annotations; the Adapters record execution behavior and failures.Source: §3.1 · p. 3

3.2. Unified Causal Execution and Declared Conditions

  • The Core controls the legal causal stream, while Adapters record actual history processing; runs declare state maintenance, response initiation, evaluation boundary, and deployment separately.Native Streaming reuses persistent state; Non-native Streaming reconstructs a causal prefix or window at query time.Source: §3.2 · pp. 3–4

3.3. Evaluation Metrics

  • QA uses only frames available by question time; accuracy requires one unambiguous valid option, while failures, timeouts, and invalid outputs remain in the denominator.Completion Rate and Invalid Output Rate separately report successful execution and responses without a uniquely recoverable option.Source: §3.3 · p. 5
  • QA Response Latency includes post-arrival history replay, preparation, queueing, and generation; TTFT diagnoses generation onset, while image submissions and output tokens describe workload, not normalized compute cost.Response Latency runs from question arrival to completion; TTFT runs from question arrival to the first output token.Source: §3.3 · p. 5
  • Proactive responses are assigned by first-token onset; In-window Accuracy scores assigned content, Median Response Delay measures onset delay, and false-alarm and miss rates track response-selection errors.A false alarm occurs when no response window is currently valid but a later opportunity remains; a miss is a target window with no assigned response.Source: §3.3 · pp. 5–6

3.4. Evaluation Scope

  • TRACE currently covers visual-only, single-instruction streaming video at 1 FPS, with proactive configurations self-initiating responses after one monitoring instruction.Execution boundaries are reported explicitly.Source: §3.4 · p. 6

4. Benchmark Construction and Temporal Audit

TRACE builds a visual-only benchmark with reviewed temporal annotations and a causal evaluation protocol. Its audit defines when evidence supports an answer or proactive response, and the standard 1 FPS set excludes a small sampling-stress subset.

4. Benchmark Construction and Temporal Audit

  • The benchmark adapts visual-only QA and Proactive Response tasks from StreamingBench and OVO-Bench into a common structure with audited temporal annotations.Records that cannot be answered from visual evidence without audio, subtitles, or external knowledge are not retained.Source: §4 · p. 6

4.1. Temporal Annotations for Causal Evaluation

  • For QA, annotators mark the earliest time visual evidence supports an answer given the question and options, no later than question arrival.The evidence-to-question distance is recorded but is not itself a difficulty measure; unanswerable questions require checking the permitted history.Source: §4.1 · pp. 6–7
  • Proactive triggers follow instruction-specific rules: onset, action completion, sufficient visible evidence, or the interval in which a state remains valid.The same action can use different triggers depending on the instruction; these categories describe trigger rules, not mutually exclusive video-content types.Source: §4.1 · p. 7

4.2. Data Preparation and Quality Control

  • Quality control combined risk screening, annotation review, targeted verification, and a published change ledger; 328 records were edited and 52 excluded.Among edited records, 325 changed a timing or trigger field; directly comparable timestamp revisions had median absolute changes of 3.0 s for QA and 2.24 s for Proactive Response.Source: §4.2 · p. 7

4.3. Final Standard Set and Sampling Observability

  • The standard 1 FPS set contains 1,240 records from 517 videos, while eight short-window Proactive records remain in a released stress subset.The released dataset contains 1,248 records across 522 videos; the standard set excludes 25 windows of at most one second that cannot be reliably sampled at 1 FPS.Source: §4.3 · p. 7

5. Experiments

TRACE evaluates systems under explicit execution conditions and reports QA and proactive outcomes across separate quality, timing, and operational measures. The results show that similar task scores can accompany different completion, output, response-selection, and timing behavior.

5. Experiments

  • The comparison includes eight public models or systems in eight configurations; failed runs remain in the relevant denominators.Core delivers timestamped RGB frames at 1 FPS with real-time pacing; timing and workload are configuration-specific, not hardware-normalized.Source: §5 · p. 7, §5.1 · p. 7

5.1. Experimental Setup and Comparison Groups

  • Table 3 makes each tested configuration’s execution conditions explicit, separating visual-state maintenance, response mode, and deployment or evaluation boundary.The comparison includes model-plus-Adapter configurations, JoyAI as a complete system, and self-initiated Proactive responses under one monitoring instruction.Source: §5.1 · pp. 7–8, Table 3 · p. 8

5.2. Main QA Results

  • LiveCC and MOSS-Preview have nearly the same QA accuracy but different completion, output-token, and invalid-output profiles; VideoLLM-Online’s low accuracy coincides with a high invalid-output rate.MOSS-VL combines higher QA accuracy with a recorded token total similar to MOSS-Preview; MiniCPM-O native duplex has 93.52% of outputs unrecoverable as a unique option under the text protocol.Source: §5.2 · p. 8, Figure 3 · p. 9, Table 4 · p. 9

5.3. Main Proactive Results

  • LiveCC combines higher window-level accuracy and near-complete target-window coverage with a high False-alarm Rate, while MOSS-VL and AURA have similar In-window Accuracy but differ in delay and false alarms.Median Response Delay is conditional on answered windows with an observed onset, so it describes observed timing rather than target-window coverage.Source: §5.3 · pp. 9–10, Table 5 · p. 10

6. Analysis and Key Findings

TRACE’s findings show that temporal validity, operational behavior, and execution conditions each shape how streaming-video results should be interpreted. Similar task scores can accompany different response and workload profiles, so the measurements remain separate.

6.1. Finding 1: Temporal Audit Establishes Reliable Measurement Boundaries

  • Temporal review edited 328 records and excluded 52; 325 edits involved timing or trigger fields, and median timestamp revisions were 3.0 s for QA and 2.24 s for Proactive Response.Reviewed evidence times determine legal visual access for QA, while audited triggers and valid intervals determine in-window status, delay, false alarms, and misses.Source: §6.1 · p. 10

6.2. Finding 2: Similar Task Scores Can Hide Different System Behaviors

  • Nearly identical QA Accuracy coincides with differences in completion, output volume, and Invalid Output Rate, while similar proactive accuracy can mask different false-alarm rates and delays.LiveCC and MOSS-Preview score 65.19% and 65.07% Accuracy, but differ in completion (93.88% versus all records), output tokens (146,064 versus 42,206), and Invalid Output Rate (6.12% versus 1.92%). MOSS-VL and AURA have 8.05% and 7.92% In-window Accuracy, with False-alarm Rates of 41.1% and 59.1% and median Response Delays of 0.33 s and 1.15 s.Source: §6.2 · pp. 10–11
  • MOSS-Preview responds earlier and misses fewer target windows than MOSS-VL, yet has lower In-window Accuracy and a higher False-alarm Rate.MOSS-Preview records 4.45% In-window Accuracy, 0.20 s median Response Delay, 77.7% False-alarm Rate, and 35.67% Miss Rate; MOSS-VL records 8.05%, 0.33 s, 41.1%, and 47.24%, respectively.Source: §6.2 · p. 11
  • Accuracy, completion, answer validity, workload, response timing, false alarms, and misses describe distinct behavior and are reported separately rather than collapsed into one score.LiveCC’s 0.31% Miss Rate coexists with false alarms in 65.0% of its assembled response episodes, illustrating that target coverage and off-window responding differ.Source: §6.2 · pp. 11–12

6.3. Finding 3: Execution Conditions Define What a Score Measures

  • History-processing paths and evaluation boundaries affect what a score measures, so identical metrics need not include the same components, workload, or failure sources.AURA reconstructs a legal visual prefix for QA but maintains persistent incremental state for Proactive Response; JoyAI’s complete-system result includes tested memory and scheduling components.Source: §6.3 · p. 12

7. Conclusion

TRACE treats streaming-video evaluation as condition-aware measurement, making temporal validity, execution conditions, and operational behavior explicit. Experiments show that these distinctions affect how results should be interpreted.

7. Conclusion

  • TRACE frames streaming-video evaluation as a measurement problem requiring explicit temporal validity, response-producing execution conditions, and operational behavior alongside task quality.The framework uses temporally audited QA evidence and proactive response windows, a controlled Core–Adapter protocol, and multidimensional reporting.Source: §7 · p. 12
  • Temporal audit revises evaluation boundaries, similar task scores can accompany different operational behaviors, and history organization or system boundaries change what scores represent.The reported dimensions include quality, timing, response selection, workload, completion, and reliability.Source: §7 · p. 12
  • TRACE aims to improve interpretability by making the conditions behind accuracy explicit and reproducible, rather than replacing task accuracy.Source: §7 · p. 12

8. Limitations and Future Directions

The findings are scoped to tested visual-only, single-instruction tasks at 1 FPS, and the benchmark prioritizes explicit temporal and execution conditions over broader modalities and interaction patterns.

8. Limitations and Future Directions

  • Current findings cover tested models or systems on visual-only, single-instruction tasks at 1 FPS.Source: §8 · p. 12
  • Proactive tasks emphasize positive triggers, so False-alarm Rate concerns responses outside a currently valid window when a later opportunity remains, not general false positives on no-trigger videos.Source: §8 · p. 12
  • Potential extensions include no-trigger and negative-event coverage, higher sampling rates, audio and multi-turn interaction, and sustained-operation tests.These extensions would broaden the operating regimes covered while retaining explicit temporal validity, execution conditions, and measurement boundaries.Source: §8 · p. 12

A.1. Released Streaming-Evaluation Projects

The appendix describes criteria for comparing released streaming-evaluation projects, details proactive-judge calibration, and explains how trigger annotations define response eligibility.

A.1. Released Streaming-Evaluation Projects

  • The comparison rubric counts a construct only when it is represented in released data or evaluators; TRACE and OmniInteract include explicit false-alarm/miss diagnostics.The surveyed projects are limited to those with public data or annotations and executable evaluation assets at the 2026-07 release audit.Source: §A.1 · p. 14

A.2. Proactive Judge Calibration

  • Human and Judge binary scores agreed on 30/36 calibrated responses (83.3%; Cohen’s kappa 0.675), but the stratified sample cannot test uniformity of bias across models.Disagreements were conservative in this sample, not evidence of equal underestimation across models or responses.Source: §A.2 · p. 14

A.3. Proactive Trigger Examples

  • Instruction-dependent trigger annotations specify when an answer becomes eligible, not task difficulty or ability levels.Point-event windows begin at the trigger, while state-task windows end at the annotated state boundary; onset and completion may mark different moments of one action.Source: §A.3 · pp. 14–15

B. Supplementary Evaluation Details and Results

The QA parsing rules define which outputs receive credit and distinguish the main recoverable parsing rule from a stricter format diagnostic.

B.1. QA Answer Parsing

  • QA Accuracy accepts a unique explicit option label, including a wrapper or label-prefixed option text, but does not semantically match unlabeled free text.Visible answers are parsed after hidden thinking spans are removed.Source: §B.1 · p. 15
  • Two distinct option labels invalidate an answer, even when one is rejected; repeating the same label does not.Ordinary articles are not treated as option labels.Source: §B.1 · p. 15
  • Strict single-label accuracy is a format diagnostic, while main QA results use the recoverable parsing rule.The examples distinguish accepted label forms from unlabeled or conflicting outputs.Source: §B.1 · p. 15, Table 7 · p. 15

B.2. Supplementary Timing, Workload, and Frame Processing

The supplementary measurements distinguish timing coverage, response and frame workload, completion, and repeated responses, while tying observations to actual execution conditions.

B.2. Supplementary Timing, Workload, and Frame Processing

  • TTFT ends at the first output token, while Response Latency ends at call completion; both include query-time history reconstruction and input preparation after question arrival.Table 8 reports recorded TTFT for QA.Source: §B.2 · p. 15, Table 8 · p. 15
  • TTFT quantiles use observed values, and coverage counts retained query groups with TTFT rather than successful QA records; missing times are not imputed.These observations do not establish end-to-end speed rankings.Source: §B.2 · p. 15
  • Median Response Delay uses only windows with observed onset times, so low coverage limits how broadly its median represents answered windows.Table 9 reports the answered and observed-onset populations behind the delay measurement.Source: §B.2 · p. 15, Table 9 · p. 15
  • Proactive workload reports input and generation volumes; image submissions include repeated history, while unique frames count distinct submitted observations.Table 10 covers 407 standard Proactive records.Source: §B.2 · p. 15, Table 10 · p. 16
  • Record completion is distinct from target-window coverage, latency-field coverage, and stream-level frame completion; the native duplex diagnostic is excluded for its different judgment path.Table 11 reports record-level completion for standard response tracks.Source: §B.2 · p. 16, Table 11 · p. 16
  • Observed measurements describe actual execution: longer response tolerances, frame grouping, queue alignment, truncation, and drops mean shared scoring populations do not guarantee identical input budgets.Some outputs were scored uniformly at five seconds despite longer response tolerances in some runs.Source: §B.2 · p. 16
  • Repeated in-window response episodes are reported descriptively: later episodes within the same strict window are redundant, but are not subtracted from accuracy or treated as errors.Table 12 reports redundant in-window episodes.Source: §B.2 · p. 16, Table 12 · p. 16
  • Frame completion and dropped-frame rates use different reporting sources and need not sum to 100%; submission or history insertion alone does not prove completed, on-time state updates.Per-frame latency and on-time rates are therefore not compared.Source: §B.2 · p. 16, Table 13 · p. 16

B.3. Trigger-Type Results and Statistical Uncertainty

Trigger-type results are presented as descriptive comparisons with video-clustered uncertainty, while response assignment uses onset timing and strict window boundaries. MiniCPM-O native duplex is reported as a diagnostic configuration under its own interaction boundary.

B.3. Trigger-Type Results and Statistical Uncertainty

  • Video-clustered 95% confidence intervals use 2,000 percentile-bootstrap resamples, keeping records from the same video together; overlapping intervals do not establish a ranking.Pairwise differences use shared video clusters for both configurations.Source: §B.3 · p. 17

B.4. Response-Segment Assignment

  • A response segment is assigned by its first-token time but judged by eventual content, so content generated after a strict-window onset cannot earn assignment to that window.Missing first-token timestamps fall back to Core receipt time and count as incomplete for latency coverage; later responses after an answer are redundant, even if it was wrong.Source: §B.4 · p. 17
  • Execution ends at the last strict-window boundary; auxiliary post-trigger scoring allows point-event credit before the next trigger, while state tasks retain their annotated end.Windows without recorded latency are excluded from Median Response Delay, and failed Judge requests are distinguished from completed content judgments.Source: §B.4 · p. 17
  • 0.51% In-window Accuracy over 1,270 target windows and 1.24 s median delay from 108 observed onsets among 439 answered windows characterize native duplex as diagnostic.Its spoken-style fragments leave 779 of 833 QA answers without a unique parse, so it is not treated as a text-interface baseline.Source: §B.4 · p. 17

B.3. Trigger-Type Results and Statistical Uncertainty

  • Trigger-type point estimates are descriptive because trigger subsets differ in size and content, rather than controlled difficulty groups.Figure 5 shows estimates for seven standard response tracks; Table 14 gives exact audited values for AURA and two MOSS configurations.Source: §B.3 · p. 16, Figure 5 · p. 17, Table 14 · p. 17
Loading 2609.30670v1…