PaperlayerView exact revision on arXiv

Source-linked AI summary

EARS: Evidence-Aware Retrieval via Sound and Motion for Budgeted Video Reasoning

Shih-Yao Lin

arXiv:2610.35536v1cs.CV

TL;DR

Uniform sampling can miss brief evidence under small frame budgets. EARS ranks frames using motion and audio, and its XD-subset result is stronger than centered Uniform at K = 1, though performance depends on the video regime.

  • Problem

    Long videos contain sparse, unevenly distributed information, and uniform sampling under limited frame budgets may miss brief critical evidence.

    Source: §1 · p. 1

  • Method

    EARS is a training-free, query-free front-end that scores frames from motion and audio, then uses temporal diversity to select frames for a downstream VLM.

    Source: §1 · p. 1, §3.4 · p. 5

  • Results

    EARS helps when anomaly evidence aligns with audio-visual cues but trails Uniform on mostly-silent UCF-Crime; at K = 1, XD-subset F1 is 0.660 versus centered Uniform’s 0.614.

    Source: §1 · p. 1, §4.2 · p. 6, §5 · p. 8

  • Takeaways & Limitations

    EARS targets tight budgets with sparse events aligned to motion or audio; Uniform catches up at larger budgets and wins on dense-narrative, audio-silent EgoSchema.

    Source: §4.7 · p. 8

    On mostly-silent UCF-Crime, EARS trails Uniform by 0.025–0.051 F1 across budgets, as subtle anomalies may lack audio or motion spikes.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

Uniform frame sampling can miss brief evidence in sparse-event videos when the VLM sees only K frames. We propose EARS (Evidence-Aware Retrieval via Sound and motion), a training-free, query-free front-end that ranks frames using optical-flow motion and audio energy, then applies temporal non-maximum suppression for diversity. The evidence score alone has frame-level AP 0.39; used only to select frames for the VLM, EARS with temporal diversity reaches localization AP 0.94, compared with 0.88 under Uniform sampling. In the updated matched study, at K = 1 EARS reaches XD-subset F1 0.660 versus 0.614 for centered Uniform (the paired interval includes zero); earlier backbone studies use separate protocols. On mostly-silent UCF-Crime, however, EARS slightly trails Uniform. A single configuration is used across both benchmarks; the new controls do not repeat the original held-out search. EARS therefore targets low-budget videos whose salient evidence aligns with motion or audio; on dense narratives or stealthy events, uniform or semantic selection remains preferable, and cheap modality statistics cannot reliably route between them. Code: https://github.com/mikeslin-ir/EARS

1. Introduction

Long-video VLM reasoning is costly, and uniform frame budgets can miss brief evidence. EARS selects frames using training-free motion–audio cues, helping in aligned sparse-event regimes but not universally.

1. Introduction

  • Long-video VLM use is computationally expensive, while uniform sampling can miss brief critical evidence under limited frame budgets.Useful video information is often sparse and unevenly distributed, so processing more frames does not ensure more meaningful evidence.Source: §1 · p. 1
  • EARS is a training-free, query-free front-end that selects high-evidence frames using motion and audio, without extra VLM passes or training.It is complementary to semantic or query-aware signals and can serve as a fallback when those are unavailable.Source: §1 · p. 1
  • EARS strongly improves XD-Violence when audio-visual evidence aligns with anomalies, but slightly trails Uniform on mostly-silent UCF-Crime and dense-narrative EgoSchema.The figure contrasts Uniform missing a brief event with EARS routing frames using motion–audio evidence under K=6.Source: §1 · p. 1, Figure 1 · p. 2
  • At K = 1, the updated study reports a +0.047 XD-subset F1 point difference for EARS against centered Uniform.The paired interval includes zero.Source: §1 · p. 2
  • Eleven modality-intrinsic statistics fail to improve per-video routing over always using EARS, and 0 of 92 tests survive correction.An oracle would gain +0.104 over always using EARS, but the tested cheap router scores below never routing.Source: §1 · p. 2

2. Related Work

Prior work addresses long-video reasoning through VLM scaling, adaptive or query-aware frame selection, token compression, and audio-visual anomaly analysis. EARS instead provides query-free, training-free motion–audio frame selection as a complementary low-cost option.

2.1. Video-Language Models

  • Long-video VLM approaches extend context or add memory, but still rely on uniformly sampled frames, whose limited budgets can hurt performance.Representative systems include VideoChat [14], Video-LLaMA [35], Video-LLaVA [16], and Qwen2-VL [25].Source: §2.1 · p. 2

2.2. Frame Sampling and Keyframe Selection

  • Adaptive selectors range from query-aware localizers to training-free semantic methods, while EARS uses cheap motion–audio cues without queries or extra VLM passes.Examples include SeViLA [31], AKS [23], VideoAgent [26], WFS-SB [7], EFS [6], and MEC [5].Source: §2.2 · pp. 2–3

2.3. Efficient Video Understanding

  • Token pruning and multimodal token compression act after encoding, whereas EARS selects frames before VLM encoding; the approaches are complementary.The cited token methods reduce tokens in the remaining representation rather than deciding which frames enter the VLM.Source: §2.3 · p. 3

2.4. Multimodal Event and Audio-Visual Analysis

  • Audio-visual anomaly methods fuse cues to produce snippet-level anomaly scores; EARS reuses motion and audio as a task-independent frame-selection front-end.The cited benchmarks include UCF-Crime [22] and XD-Violence [28].Source: §2.4 · p. 3

2.5. Summary

  • EARS combines training-free, single-pass, query-free audio-visual selection, with scores additive to semantic or query-aware signals and usable as a cheap default.Prior work commonly scales VLMs, trains query-aware selectors, or fuses audio and vision to predict anomaly labels.Source: §2.5 · p. 3

3. Method

EARS approximates downstream frame-selection goals with a cheap multimodal evidence proxy, then selects high-evidence frames while enforcing temporal diversity. Its motion and audio signals are label-agnostic; the selection procedure is designed to reduce redundant frames.

3.1. Problem Setup

  • EARS approximates selecting K frames to maximize downstream VLM performance by using a cheap, training-free proxy of multimodal information density.The pipeline extracts motion and audio signals, fuses them into an evidence score, and selects frames with temporal diversity.Source: §3.1 · pp. 3–4

3.2. Multimodal Signal Extraction

  • Dense optical flow supplies per-frame motion magnitude, while timestamp-aligned audio windows supply acoustic energy; silent or corrupted clips use a motion-only variant.Flow magnitude is less sensitive to illumination changes than raw frame difference, though global camera motion can still create spurious evidence.Source: §3.2 · p. 4

3.3. Evidence Score

  • Per-video min-max normalization puts motion and audio signals on a common [0, 1] scale before their linear fusion into a label-agnostic evidence score.The score indicates modality activity, not what event occurred; the VLM determines what the evidence implies.Source: §3.3 · p. 4

3.4. Evidence Selection with Information Compression

  • Figure 2 shows why vanilla top-K can cluster around one peak; IC instead greedily suppresses nearby frames using a minimum gap τ.Smaller τ behaves more like vanilla top-K, while larger τ spreads selections farther apart; the resulting subset is passed to the VLM, with fusion and IC adding O(T) work.Source: §3.4 · pp. 4–5, Figure 2 · p. 5

4. Experiments

The experiments test EARS on sparse-anomaly surveillance benchmarks under constrained frame budgets, comparing it with Uniform and query-aware selectors. Results show that EARS’s advantage depends on audio–motion alignment and budget, with qualitative wins and failures matching that scope.

4.1. Datasets, Tasks and Implementation

  • The evaluation covers XD-Violence and UCF-Crime anomaly classification, reporting Accuracy, Precision, Recall, and F1, plus frame-level AP and AUC for localization.The updated study deduplicates UCF-Crime to 290 unique test videos; earlier tables counted 440 entries including repeated normal videos.Source: §4.1 · p. 5

4.2. Anomaly Classification under Varying Frame Budgets

  • At K = 1, updated XD-subset F1 is 0.660 for EARS versus 0.614 for centered Uniform; all six paired intervals include zero.The updated matched study is separate from the original-protocol curves and tables.Source: §4.2 · p. 5, §4 · p. 5
  • On mostly-silent UCF-Crime, EARS trails Uniform by 0.025–0.051 F1 across budgets, while increasing K improves both methods.The authors link the gap to subtle anomalies such as Shoplifting and Stealing that may produce no audio or motion spike.Source: §4.2 · p. 6

4.3. Qualitative Analysis

  • In two XD-Violence examples, EARS selects frames showing a raised handgun and an on-ice brawl that Uniform misses.In the shooting clip, EARS selects four crisp close-range frames; in the hockey clip, it focuses on a roughly 30-second event window.Source: §4.3 · p. 6, Figure 4 · p. 7

4.4. Comparison with Prior Frame-Selection Methods

  • At K = 1 on XD-Violence, AKS and SigLIP top-K score 0.86 versus EARS at 0.72; EARS catches them by K = 4.The query-aware baselines use a task-specific anomaly query; on mostly-silent UCF-Crime they lead at K ≤ 2, Uniform leads at K = 4, and EARS trails both.Source: §4.4 · pp. 6–7, Table 2 · p. 7

4.5. Matched Semantic Fusion

  • On XD-Violence at four frames, SigLIP/EARS score fusion raises F1 from 0.847 to 0.889, below Uniform, EARS, and Audio at 0.893, 0.898, and 0.907.The study averages normalized SigLIP and EARS scores before applying identical NMS.Source: §4.5 · p. 7

4.6. Additional Validation

  • On XD, EARS exceeds five-seed Random by 0.05–0.10 F1 for K ≤ 4; adaptive stopping retains UCF F1 0.419 with 1.67 frames/video versus 0.427 with 3.80.The UCF adaptive-stop setting cuts frames by 56% for a 0.008 F1 difference; on XD, F1 stays approximately 0.67 as the realized budget falls from 2.42 to 2.17 frames/video.Source: §4.6 · pp. 7–8, Table 8 · p. 8

4.7. Discussion

  • A single EARS configuration targets tight budgets when sparse events align with motion or audio; Uniform catches up as K grows and wins on dense-narrative, audio-silent EgoSchema.The discussion says stealthy events may require semantic evidence and that cheap signals cannot reliably route between these regimes.Source: §4.7 · p. 8

5. Conclusion

EARS is a training-free, query-free frame router whose motion-and-audio evidence can improve low-budget localization when those signals align with events, but its gains are not universal.

5. Conclusion

  • At K = 1 on the XD subset, EARS reaches F1 0.660 versus centered Uniform’s 0.614, although the paired interval includes zero.The raw proxy has AP 0.393, while EARS+IC+VLM reaches frame-level AP 0.944.Source: §5 · p. 8
  • EARS trails Uniform on mostly-silent UCF-Crime, and dense-narrative EgoSchema requires semantic evidence.The paper positions Uniform or query-aware selection as safer for silent, semantic, or dense-narrative videos.Source: §5 · pp. 8–9
  • A cached, query-independent evidence signal can be reused across budgets and future questions, making EARS most useful for repeated reasoning over motion- or audio-correlated events.The deployment trade-off is a one-time CPU preprocessing cost.Source: §5 · p. 9

Supplementary Material

The supplementary material extends the paper’s empirical evaluation and provides additional procedural detail.

Supplementary Material

  • The supplementary reports random-seed analysis, out-of-domain evaluation, and backbone transfer.It also covers extended comparisons and frame-level localization.Source: Supplementary Material · p. 10
  • It includes ablations, efficiency measurements, adaptive-budget analysis, and a per-video routing study.The listed analyses extend the experimental results beyond the main paper.Source: Supplementary Material · p. 10
  • Main-paper Fig. 5 visualizes characteristic failure modes, while Algorithm 1 gives the complete Information Compression procedure.Source: Supplementary Material · p. 10

S1. Additional Experimental Results

The supplementary tests EARS across benchmarks, backbones, design choices, and deployment settings. The results support a conditional sparse-event use case, while identifying limits to semantic generality and per-video routing.

S1. Additional Experimental Results

  • Random-7B seed standard deviations are ≤0.027 F1, while EARS exceeds Random by 0.05–0.10 F1 across XD K ≤4; on UCF, both trail Uniform.The random baseline uses five seeds under the same backbone, prompt, and split; EARS and Uniform are deterministic.Source: S1. Additional Experimental Results · p. 10, Table S2 · p. 10
  • On the EgoSchema 500-question subset, Uniform outperforms EARS by 5.6–9.0 accuracy points at K = 4 and K = 8.The benchmark is dense-narrative and audio-silent; the paper frames EARS as complementary to semantic or query-aware selectors.Source: S1. Additional Experimental Results · pp. 10–11, Table S4 · p. 11

S1.1. Backbone Generality

  • Across four backbones, XD-Violence K = 1 F1 improves by +0.34–+0.43 with EARS, though UCF-Crime is not a general gain.The additional-backbone study swaps only the VLM; the reported cross-family LLaVA evidence is limited to XD at K ≤4, where that backbone is competent.Source: S1.1. Backbone Generality · pp. 11–12, Table S5 · p. 12

S1.2. Frame-Level Temporal Localization

  • On frame-level XD-Violence localization, the raw evidence score is a weak detector but a useful VLM selector, with Information Compression adding further AP.The table caption reports raw proxy AP = 0.393, a +6.3 AP lift over Uniform from routing, and a further +1.9 with Information Compression.Source: Table S6 · p. 12, S1.2. Frame-Level Temporal Localization · p. 12

S1.3. Ablations

  • A single held-out configuration is used across datasets, while ablations show that fusion, temporal diversity, and flow choice have dataset-dependent effects.Balanced fusion yields XD F1 = 0.688 and stays within 0.017 F1 of motion-only on UCF; a 0.5 s IC gap lifts UCF F1 by +0.077, whereas τ = 0 is optimal on XD. TV-L1 and Farneback results stay within 0.024 F1 at tight XD budgets.Source: S1.3. Ablations · pp. 12–13

S1.4. Efficiency Analysis

  • On 24 XD-Violence and UCF-Crime videos, preprocessing took 3.12 s/video; at K = 4 on 24 XD videos, EARS took 6.87 ± 2.95 s/video end to end versus 1.33 ± 0.53 for Uniform, so EARS was slower initially.Its query-independent preprocessing cost is cached and amortized across later queries.Source: S1.4. Efficiency Analysis · p. 13

S1.5. Can Selection Be Routed Per Video?

  • The leave-one-dataset-out per-video router scores 0.635, below the 0.647 constant policy, despite an oracle margin of +0.104 over always-EARS.The learned threshold collapses to a constant arm; 79–92% of videos receive identical predictions under both arms, and the oracle gap remains unresolved.Source: S1.5. Can Selection Be Routed Per Video? · pp. 14–15, Table S9 · p. 14

S1.6. Discussion

  • EARS suits tight budgets when sparse events align with motion or audio, but stealthy events without a motion or audio spike require a semantic channel.The discussion also describes a selectivity-annealed hybrid between evidence-driven and uniform sampling; extending to dense-narrative suites is tied to adding semantic evidence.Source: S1.6. Discussion · p. 15

S2. Frame-Level Curves

The supplementary figure presents two types of frame-level XD-Violence evaluation curves.

S2. Frame-Level Curves

  • Figure S3 plots precision–recall and ROC curves for frame-level XD-Violence annotations.Source: Figure S3 · p. 15
  • The left panel is the precision–recall curve for the frame-level annotations.Source: Figure S3 · p. 15
  • The right panel is the ROC curve for the same frame-level annotations.Source: Figure S3 · p. 15

S3. Updated Matched Controls

The updated matched controls correct video identities, labels, and the Uniform baseline, and are separate from the original studies. They compare EARS with semantic and simple alternatives across two cohorts, while keeping the updated results distinct from earlier protocols.

S3. Updated Matched Controls

S3.1. Cohorts, Reasoner, and Signal Alignment

  • The XD subset contains 120 Fighting, 84 Shooting, and 300 Normal videos; UCF contains 140 anomalous and 150 normal unique test videos.The XD data are an available local test subset, not the full official benchmark; 222 UCF audio signals have identically zero amplitude, which does not prove stream absence.Source: S3.1. Cohorts, Reasoner, and Signal… · p. 15

S3.2. Semantic Fusion and Simple Alternatives

  • 0.016 on XD and 0.037 on UCF: semantic NMS improves four-frame F1 over plain top-K, with exploratory intervals [0.001, 0.033] and [0.009, 0.069].The reported change must not be attributed to activity fusion; all six Difference-minus-EARS intervals include zero, establishing neither flow superiority nor proxy equivalence.Source: S3.2. Semantic Fusion and Simple… · p. 16

S3.3. Paired Cost and Cache Reuse

  • Fresh-feature EARS has lower marginal medians than semantic selection but higher total cost than Uniform; cached EARS-minus-Uniform intervals include zero.Thus, no reliable cached speedup or equivalence is established; the hybrid, semantic-NMS, and difference selectors lack a matched timing study.Source: S3.3. Paired Cost and Cache Reuse · p. 16

Related papers

Loading 2610.35536v1…