Source-linked AI summary
Pre-Decoding Acoustic Triage for Budgeted Vision-Language Captioning of Untrimmed Egocentric Video
Masoud Jalayer, Changyi Li, Yu Xiao
TL;DR
Repeated VLM calls make hours-long egocentric-video analysis costly, while visual triage requires decoding the video it is meant to save. The paper uses pre-decoding audio-first triage with a span-level, once-per-action objective and spaced call selection. It reports improved action coverage across budgets, competitive benchmark results, and low acoustic-screening cost, while requiring annotated target-action spans and leaving most quantitative evidence on EK-100.
Problem
Hours-long egocentric-video captioning is expensive because VLM cost scales with repeated calls, and visual selectors require video decoding before triage.
Method
The method scores audio before video decoding using frozen AudioSet-pretrained features, trains a span-level gate to fire once per action, and spaces selected windows under a VLM-call budget.
Results
4.0–10.8 percentage points of action-coverage improvement over frame-level cross-entropy holds at every operating point, while spaced selection improves coverage where calls are scarce.
Takeaways & Limitations
Pre-decoding acoustic triage reduces the number of VLM calls while composing with per-call efficiency methods and outperforming uniform sampling and recent visual selectors in reported settings.
Takeaways & Limitations
Most quantitative evidence comes from EK-100, VLM analyses use a smaller subset, and the temporal head still requires annotated target-action spans.
Abstract
from arXiv · showhide
Automatically analyzing hours-long egocentric video is increasingly essential for progress monitoring, quality control, and safety in logistics, construction, and manufacturing. Yet current pipelines that process short, fixed-size windows with a vision-language model (VLM) are prohibitively expensive because cost scales with the number of model calls. To reduce this cost, prior work proposes triage policies to select which windows merit a VLM invocation. However, these policies either sample uniformly or rank windows using visual features, which ironically requires the video decoding that the budget constraints are meant to avoid. We propose audio-first triage: select windows using the lightest modality, scored before any video frame is decoded, so the approach composes naturally with token compression or quantization. The novelty lies in the objective, not the representation: rather than a per-frame sound-event detector, we train the selector to trigger once per action. This objective shift improves action coverage by 4.0-10.8 percentage points across all evaluated call rates, using frozen AudioSet-pretrained features without domain-specific sound-event labels. Using fewer than half of the available calls, the triage cuts 9-20% of VLM calls at matched coverage on EPIC-KITCHENS-100 (EK-100), surpasses uniform sampling through the mid-range on Ego4D over 247 clips, and outperforms two recent visual keyframe selectors. Code, the reference implementation and every results file this manuscript reads are at https://github.com/masjalayer/PreDecoding-AcousticTriage.
1 Introduction
The paper targets the cost of repeatedly sending short windows from hours-long egocentric recordings to a VLM. It proposes audio-first, pre-decoding triage trained to select sparse evidence for distinct actions under a call budget.
- 185 GPU-hours can be required to forward every four-second window of a 1,000-hour archive to an 8B model.
- Audio-first triage scores windows before video decoding, avoiding the visual processing required by content-aware selectors.The approach uses audio because it is cheaper to process and compatible with visual token compression or weight quantization.
- Span-level multiple-instance learning triggers once per action rather than marking every frame of a long action.The selector uses frozen AudioSet-pretrained features and penalizes activity outside annotated action spans.
- 4.0 to 10.8 points of action coverage improvement over frame-level cross-entropy occurs at every evaluated budget.
- 46.8 ± 1.1% of actions are recovered versus 40.9% at 25% of calls, with gains over uniform sampling on 22 of 24 EK-100 recordings and through the mid-budget range on 247 Ego4D clips.
- The paper notes that call-based costing can undercharge its own gate by 1.76× under a retained-duration account, while high occupancy limits triage gains.Measured occupancy ranges from 85.1 to 99.3% across datasets.
2 Related Work
Prior work reduces egocentric-video understanding costs through visual selection, audio detection, or per-call VLM efficiency, but generally makes selection after visual processing. This paper moves selection before decoding and aligns audio supervision with distinct-action coverage.
- Egocentric video understanding: Egocentric-video research spans activity recognition, temporal localization, multimodal reasoning, and large-scale benchmarks such as EK-100 and Ego4D.
- Sound event detection and MIL: AudioSet-pretrained representations support sound-event detection, while this work evaluates PANNs and AudioSet-Strong features including BEATs, ATST-Frame, and fPaSST.
- Sound event detection and MIL: Unlike weakly supervised audio tagging, this setting uses strong action spans as positive bags and requires evidence somewhere within each span.The criterion matches a downstream objective where one selected window can cover an action.
- Frame selection under call constraints: Visual frame selectors such as AKS, MDP3, BOLT, Q-Frame, and Frame-Voyager allocate fixed frame budgets using relevance, diversity, or query-aware signals.
- Audio-based and audio-visual selection: Chapter-Llama uses speech transcripts, VFSTA uses audio-event cues with visual processing, and audio-visual localization methods jointly process decoded visual input.
- Reducing VLM inference cost: Per-call efficiency methods prune visual tokens or skip redundant computation, making them complementary to a method that reduces the number of VLM calls.
- The paper’s distinction is pre-decoding audio scoring, distinct-action coverage training, and allocation under an explicit VLM-call constraint.
3 Method
The method defines action coverage and cost directly over fixed VLM-call windows, then uses a frozen audio representation, a sparse span-level gate, and spaced window selection. Only selected windows are decoded and sent to the VLM.
- 3.1 Budgeted action coverage: An exhaustive scan over a recording of duration T uses M = ⌈T/w⌉ calls, while triage retains at most Kmax = ⌊ρM⌋ windows under budget ρ.
- 3.1 Budgeted action coverage: Annotated action intervals score coverage but are unavailable at inference, and all methods are compared at matched call budgets.
- 3.1 Budgeted action coverage: Calls are the cost unit because every selected window has the same length, frame count, token allowance, and prompt.The measured operating-point cost is 0.74 GPU-seconds per call.
- 3.1 Budgeted action coverage: A retained-duration account undercounts the gate’s 2,109 calls as 1,201 equivalent calls, producing a 1.76× discrepancy.
- 3.1 Budgeted action coverage: Because coverage is unchanged after the first useful call but cost grows linearly with calls, the objective favors sparse evidence that touches every action.Temporally correlated scores also require selection to spread calls rather than merely rank them.
- 3.2 Audio representation: The frozen BEATs AudioSet-Strong frontend produces logit-space features for the gate, with 10-second chunks and 5-second hops mapped onto a 25 Hz global grid.
- 3.2 Audio representation: Freezing the encoder keeps the frontline cheap and requires only annotated target-action spans rather than a domain-specific sound taxonomy.
- 3.3 Coverage-aligned acoustic gate: The span-level gate uses a small three-block dilated temporal CNN to map audio features to frame-level activity logits.Its dilations are 1, 4, and 16, with 0.41M parameters.
4 Experimental Results
Experiments across EK-100 and Ego4D show that occupancy limits triage headroom, while score-based selection becomes effective when paired with temporal spacing. The proposed rule improves coverage or reduces calls below half-budget settings, but loses its advantage as calls become plentiful and remains sensitive to evaluation and accounting choices.
- 4.2 Occupancy of the stream: 85.1% occupancy on EK-100 four-second windows leaves uniform sampling close to optimal and limits content-aware triage headroom.The dataset averages 1.41 annotated actions per window.
- 4.2 Occupancy of the stream: −0.59 Pearson correlation links lower occupancy to greater triage gain at 5% of calls, but the relationship vanishes by half the stream.The authors treat occupancy as a headroom estimate rather than a general law.
- 4.3 Selecting which windows to call: 9% to 20% fewer calls than uniform at matched coverage occurs below half the stream, but savings turn negative above half.This call-based accounting differs from retained-duration accounting, which can overstate savings for fragmented selections.
- 4.3 Selecting which windows to call: 6.0 points and 5.6 points gains over uniform on participant-disjoint and participant-overlap subsets preserve the ordering below half the stream.The replication set contains longer actions, narrowing but not closing the margin.
- 4.3 Selecting which windows to call: 5.8 points average gain over uniform sampling at 25% of calls occurs on 22 of 24 pooled recordings, while BOLT remains closest.The paired comparison is statistically significant for the proposed rule and also compares it with AKS and rank-only selection.
- 4.4 The training objective, and what transfers: 53.9 ± 7.3% stream discarded at comparable recall versus 31.5% for the AudioSet tagger reflects the span-level objective’s advantage, not necessarily call savings.The retained-duration comparison is explicitly treated as partly an artifact of the accounting unit.
5 Discussion
Pre-decoding triage is most useful when target activity is sparse, acoustically observable, and cheap to screen, but dense datasets leave limited headroom beyond uniform sampling. The evaluation also shows that call-based accounting and temporal alignment materially shape conclusions.
- Cost and opportunity: 3% of an exhaustive VLM scan is the acoustic screening cost, versus 75% for optical flow.The acoustic gate remains below the break-even threshold across the evaluated operating range.
- Cost and opportunity: 1.76× is the gate’s discrepancy when retained duration underestimates fragmented VLM calls.VLM calls are therefore the primary cost measure, with duration-based results retained as secondary.
- What makes triage effective?: Span-level supervision improves action coverage across all tested frozen audio representations, while extractor differences remain small relative to seed variation.Only the lightweight 0.41 M-parameter temporal head is trained.
- What makes triage effective?: Minimum-separation selection combines coverage-aligned scores with temporal diversity, whereas direct ranking repeatedly selects correlated evidence.The hard constraint is most useful when calls are scarce or moderate and becomes restrictive when calls are plentiful.
- What makes triage effective?: Frame- and clip-level detectors can retain most of the stream despite high recall, because the deployment objective requires one useful call within each action.The paper’s gate is optimized for efficiency under a call limit rather than dense event response.
- Limitations and scope: Quiet, weakly audible, or noise-masked manipulations may receive low gate scores despite visual salience.Lightweight fallback cues or audio-triggered adaptation are identified as possible next steps.
- Limitations and scope: The reported coverage criteria do not measure whether the downstream VLM interprets selected evidence correctly.The study counts overlap with an annotation or, under a stricter criterion, whether the representative frame falls inside the interval.
6 Conclusion
The paper evaluates pre-decoding audio triage for long egocentric video by aligning learning and cost measurement with distinct-action coverage under finite VLM calls. Span-level supervision improves coverage across operating points, while practical benefit depends on sparse activity, informative acoustics, and inexpensive screening.
- 6 Conclusion: Pre-decoding audio triage combines a span-level acoustic gate with call-limit-aware selection and evaluates cost directly in VLM calls.The design targets distinct-action coverage under a finite number of calls.
- 6 Conclusion: 4.0–10.8 percentage points is the action-coverage improvement from span-level supervision over frame-level cross-entropy at every operating point.The improvement holds across tested frozen audio representations without domain-specific sound-event labels.
- 6 Conclusion: Pre-decoding audio triage pays most when target activity is sufficiently sparse, acoustic evidence is informative, and screening remains inexpensive.Dense public datasets leave limited headroom beyond uniform sampling, while duration accounting can understate fragmented calls.
A The paired comparison at every call rate
The paired sweep tests selection policies across call rates on 24 EK-100 recordings and 247 Ego4D clips. The advantage persists across tight budgets, but uniform sampling wins when a hard separation constraint starves the selector near half the stream.
- A The paired comparison at every call rate: 24 EK-100 recordings and 247 Ego4D clips are evaluated across the full call-rate sweep using paired Wilcoxon signed-rank tests.Every rule runs on the same material at the same allowance.
- A The paired comparison at every call rate: 33.5% of calls down to the tightest tested rate is the range over which the advantage holds against uniform sampling.The advantage also holds against rank-only and AKS over the same range with comparable margins.
- A The paired comparison at every call rate: 20 of 24 recordings favor uniform sampling at half the stream, marking the largest reversal in either table.As the allowance approaches the number of admissible windows, hard separation forfeits calls it cannot place.
- A The paired comparison at every call rate: BOLT is the closest comparator throughout the paired comparison.The paper attributes this proximity to BOLT spreading selections by construction rather than by constraint.
B Every recording, at a quarter of the calls
The per-recording analysis makes the quarter-budget result visible against recording and action statistics rather than only reporting an aggregate mean. The margin is tied more closely to occupancy under scarce calls than to recording length or mean action length.
- B Every recording, at a quarter of the calls: 5.8 points over 24 recordings does not reveal whether the margin is broad or concentrated in a few streams.Table 7 reports every recording at 25% of calls alongside length and action statistics.
- B Every recording, at a quarter of the calls: Occupancy, especially when calls are scarce, tracks the margin more than recording length or mean action length.The two recordings lost by the method are associated with longer actions and higher uniform-sampling coverage, producing a narrower replication-set margin.
C Does a shorter window restore the advantage?
Shorter windows reduce occupancy but do not restore ranking’s advantage over uniform sampling. The evidence instead supports spreading calls across acoustically active stretches, regardless of window length.
- Motivation: 85.1% occupancy at w = 4 s on EK-100 means shorter windows can create sparsity, but the same action stretches are subdivided into more windows.The shorter-window hypothesis motivates testing whether ranking can exploit the resulting sparsity.
- Results: 17 points behind uniform sampling, ranking performs worst at 0.5 s and a quarter of the calls, where the stream is emptiest.The deficit closes as windows lengthen, so lower occupancy alone does not restore ranking’s advantage.
- Mechanism: Shortening the window subdivides acoustically active stretches into multiple highly scored windows that ranked selection exhausts before looking elsewhere.This produces sparsity that a score cannot exploit.
- Conclusion: Spreading calls, rather than merely shortening windows, is the selection-rule change that recovers the advantage.The conclusion applies across the tested window lengths.
D Operating points and training configuration
The pipeline’s operating constants are selected on development recordings and then fixed for evaluation, with hysteresis thresholds tied to development recall targets.
- Operating points: Thresholds and separation are chosen on development recordings and applied unchanged to evaluation recordings.No listed constant is tuned on data later used for scoring.
- Training configuration: Three hysteresis thresholds correspond to the three development recall targets reported throughout.