PaperlayerView exact revision on arXiv

Source-linked AI summary

CIViC: Evidence Gates for Audio-Conditioned Visual Prediction on Proxy Tasks

Mengzhe Geng, Max W. Y. Lam, Yuewen Cao, Shoukang Hu, Jing Xu, Marc Tessier, Xu Li

arXiv:2610.15327v1cs.LGcs.CVcs.SDeess.AS

TL;DR

Audio removal or perturbation can change a visual model’s output without showing that its task uses the intended audio information. CIViC maps conditional interpretations to four evidence gates and applies them to three controlled proxy studies. The studies produce distinct findings across output activation, held-out error, and endpoint controls, not a general claim of audio-conditioning benefit or failure.

  • Problem

    Audio-path interventions can change model outputs while leaving open whether the task uses the intended audio information.

    Source: §1 · p. 1

  • Method

    CIViC links a conditional interpretation to four diagnostic gates and fixes the split, intervention, score, aggregation unit, and decision rule before held-out scoring.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    Across the proxy studies, CREMA-D has a small static effect, the synthetic condition gap shrinks with training, and SpeakerVid favors five-feature-zero on raster RMSE but aligned audio on segment drift.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Output activation, held-out error, and endpoint controls support different interpretations across these proxy tasks, rather than a general audio-conditioning benefit or failure claim.

    Source: §1 · p. 2, §6 · p. 9

    The findings are limited to the tested proxy settings and do not establish general audio-conditioning benefit or failure.

    Source: §1 · p. 2

Abstract

from arXiv · show

Removing audio can shift a visual predictor's output without showing whether the task uses the intended audio information. We introduce Conditional Input and Visual Interpretation Checks (CIViC), a four-gate protocol that links an interpretation to an intervention, held-out task comparison, or endpoint control. We demonstrate CIViC on actor-disjoint face-raster tests, a synthetic audio-feature mixture, and source-video-disjoint SpeakerVid proxy tasks. In the static face-raster test, cross-actor audio replacement meets the registered detection rule with a small effect; on a separate temporal endpoint, constant-audio fits score better than audio-trained fits in four matched seeds. In the synthetic mixture, the condition-augmented fit has higher 1,000-step RMSE than the audio-only fit (0.1807 vs. 0.1707), with the gap shrinking during training. In the SpeakerVid proxy, the five-feature-zero fit (five low-level audio summaries zeroed; identity, dialogue, and condition inputs retained) has lower whole-frame grayscale RMSE than aligned audio across four held-out conditions and five paired seeds, while aligned audio has lower segment-drift error. The results show that output activation, held-out error, and endpoint controls support different interpretations across these proxy tasks.

1 Introduction

Audio-path interventions can change outputs without showing that a model uses the intended audio information for its task. CIViC maps a conditional interpretation to matched evidence checks, and the proxy studies yield distinct, limited findings rather than a general claim of audio-conditioning benefit or failure.

1 Introduction

  • CIViC fixes the split, intervention, score, aggregation rule, and decision rule before held-out scoring, tying evidence to a specified interpretation.The framework supplements rather than replaces common evaluation actions; Table 1 presents those actions as complementary, not ranked.Source: §1 · pp. 1–2, Table 1 · p. 2
  • The framework is a bounded interpretation-to-control mapping, not a general evaluation standard or a claim that its four checks are necessary or sufficient beyond tested settings.The studies cover actor-disjoint face-raster checks, a synthetic feature-mixture task, and a source-video-disjoint SpeakerVid proxy.Source: §1 · pp. 1–2
  • CIViC separates input activation, substitution specificity, held-out benefit, and endpoint validity into four diagnostic questions.Its case studies add no new architecture and do not rank generators or assess perceptual outputs.Source: §1 · pp. 1–2
  • The studies report distinct proxy outcomes: CREMA-D shows a small static effect, the synthetic condition gap shrinks with training, and SpeakerVid favors different fits on raster RMSE and segment drift.The results do not establish general audio-conditioning benefit or failure.Source: §1 · p. 2
  • The three-seed Study II direction has no held-out-case interval or population inference, and repeated seeds do not add independent actors or source videos.Zero cross-split overlap establishes split provenance, not a population interval.Source: §1 · pp. 2–3
  • A detection excludes only its named alternative; practical interpretation also requires a task-scale threshold or equivalence region registered in advance.CREMA-D has no practical threshold for its tiny cross-actor contrast.Source: §1 · pp. 2–3, Table 2 · p. 3

2 CIViC maps interpretations to four evidence questions

CIViC links each intended conditional interpretation to the evidence needed to distinguish it from alternatives. The gates are selected for that interpretation, with the design and decision rules fixed before held-out scoring.

2 CIViC maps interpretations to four evidence questions

  • A change between audio-present and audio-removed predictions establishes output activation only, not that the task uses the intended audio information.The four gates target alternative explanations, and studies run those required by their intended interpretation.Source: §2 · p. 3, Figure 1 · p. 3
  • Speaker-specific claims require registered speaker-audio substitution, mixed-condition benefit requires matched task comparison, and proxy transfer requires endpoint control against non-audio shortcuts.An endpoint warning excludes the named interpretation but does not identify its shortcut.Source: §2 · p. 3
  • Figure 1 contrasts CREMA-D’s fixed-pair detection without a practical threshold with synthetic condition-driven prediction changes alongside higher early held-out error.The synthetic error gap shrinks with training; its matched endpoint control is untested, and the newer SpeakerVid comparison is described elsewhere.Source: Figure 1 · p. 3
  • CIViC fixes the split, intervention, score, aggregation unit, and decision rule before held-out scoring.The gates are selected according to the intended interpretation rather than treated as a universal checklist.Source: §2 · p. 3

3 Study I: static and temporal face-raster checks

The static CREMA-D test met its registered detection rule with a very small cross-actor effect, while separate temporal and external checks did not establish broader audio benefit or speaker attribution.

3.1 CREMA-D meets a narrow fixed-pair detection rule

  • 0.000109 MAE was the CREMA-D cross-actor-audio effect; it met the registered sign-based detection rule, but practical relevance remained unclassified.No task-scale threshold was registered for this effect.Source: Table 3 · p. 4, §3.1 · p. 4

3.2 A separate temporal renderer fails its audio controls

  • Aligned MAE was 0.030449; shifting audio raised it by 0.0000176, while repeating the identity raster lowered it by 0.010592, without establishing sequence-level audio benefit.The check did not measure perceptual video quality.Source: §3.2 · p. 4

3.3 A separate confirmation fails the audio-replacement gate

  • RAVDESS replacement changed MAE by −0.00045 (95% interval [−0.00129, 0.00039]); constant-audio scored 0.08652 versus 0.08990 for audio-trained, and replacement failed.Constant-audio fits scored better in all four matched seeds.Source: §3.3 · p. 4

3.4 An incompatible external audit tests a separate boundary

  • AVA shared-soundtrack AUROC was 0.63678 aligned, 0.63670 shifted, and 0.63672 with zero audio; this check supplies no speaker-attribution evidence.The aligned-minus-shifted interval was [−0.00017, 0.00028].Source: §3.4 · p. 5

4 Study II: an active condition offers no stable benefit on a withheld feature mixture

In the synthetic feature-mixture study, adding the condition worsened early held-out error, with the gap shrinking during fitting; endpoint controls also caution against interpreting favorable scalar-proxy orderings as intended audio use.

4.1 Controlled composition protocol

  • The study used 819 synthetic mixtures from 205/24/27 source clips, with zero split overlap in source clips, original-video groups, video-namespaced speakers, and exact target arrays.Both arms retained five audio summaries, identity, and dialogue inputs; the added seven-bit condition increased input width and the joint mixture was withheld from training. The 128-coordinate grayscale target is a dimensionless proxy, not a physical motion or video-quality measure.Source: §4.1 · p. 5

4.2 The early error gap shrinks with training

  • At 1,000 steps, adding the condition raised withheld-composition RMSE in all three seeds; by 10,000 steps the mean gap was 0.0019 ± 0.0032, with one negative seed.Zeroing the condition changed predictions by 0.2123 ± 0.0070; that displacement is not a task-benefit comparison.Source: §4.2 · p. 5, Table 5 · p. 6, Figure 2 · p. 6

4.3 A parameter-matched audit exposes a training-support mismatch

  • Under clean-plus-environment support, parameter-matched policies scored 0.1822 ± 0.0025 (dummy), 0.4248 ± 0.0217 (shuffled), and 0.5003 ± 0.0278 (five-feature-zero); these controls do not isolate condition benefit.Each policy used seven coordinates and 4,908,416 parameters.Source: Table 6 · p. 7, §4.3 · p. 7

4.4 A condition-only comparator raises an endpoint warning

  • In source-disjoint scalar-proxy follow-ups, condition-only reached RMSE near 0.00109, below structured and audio-only models, so favorable ordering did not support the intended audio-conditioning interpretation.These checks do not identify the shortcut in the primary feature-mixture comparison.Source: §4.4 · p. 7

5 Study III: a source-video-disjoint SpeakerVid proxy study

Study III evaluates a source-video-disjoint SpeakerVid proxy with raster reconstruction as its primary endpoint. Five-feature-zero has lower raster RMSE than aligned audio, but aligned audio has lower segment-drift error, so the ordering is endpoint-specific.

5 Study III: a source-video-disjoint SpeakerVid proxy study

  • The source-video-disjoint study uses 865 two-speaker windows from 31 videos, with zero source-clip, source-video, and exact-target overlap between splits.The train, validation, and test records span 23, four, and four videos, respectively.Source: §5 · p. 7
  • The 128-value visual target is an 8-by-16 whole-frame grayscale raster, paired by frame index because presentation timestamps are absent.It is not a face or mouth crop, landmarks, or a semantic motion representation.Source: §5 · p. 7
  • Five-feature-zero sets five low-level audio summaries to zero but retains identity, dialogue, and aligned condition inputs, so it is not a silent-input condition.The retained condition encodes synthetic speech presence and type.Source: §5 · p. 8

5.1 Five-feature-zero has lower test error across platforms and source groups

  • Five-feature-zero has lower test raster-reconstruction RMSE in all 16 platform-condition cells, while aligned audio has lower segment-drift error in all 16.All five paired seeds are negative in each raster-RMSE cell; the four platform runs reuse one source pack.Source: Table 7 · p. 8, §5.1 · p. 8
  • Audio-only has lower mean test RMSE in all 16 cells, but omitting the condition branch changes input width; dummy conditions worsen RMSE and shuffled-condition means are small and variable.The audio-only comparison therefore is not an isolated audio effect.Source: §5.1 · pp. 8–9
  • The endpoint supports only a raster-reconstruction ordering: segment drift reverses it, and neither proxy establishes rendered-video quality or synchronization performance.The target uses whole frames, and frame-index differences without presentation timestamps establish neither temporal motion nor audio alignment.Source: §5.1 · p. 9
  • Clean-test mean RMSE is 0.2005 for aligned audio and 0.1864 for five-feature-zero, versus 0.3477 for the coordinate-wise mean baseline; this does not isolate audio.The baseline comparison is descriptive under a shifted target distribution, and five-feature-zero retains identity, dialogue, and aligned-condition inputs.Source: §5.1 · p. 9

6 Conclusion

Across the proxy studies, output activation, held-out error, and endpoint controls support different interpretations. The findings remain bounded to their specific tasks and endpoints, not a general claim of audio-conditioning benefit or failure.

6 Conclusion

  • Output activation, held-out error, and endpoint controls support different interpretations across these proxy studies, rather than one general audio-conditioning verdict.The studies do not establish general audio-conditioning benefit or failure.Source: §6 · p. 9
  • CREMA-D’s cross-actor effect is small and utterance-confounded, while Study II’s early fit gap narrows with training and condition replacements show displacement, not task benefit.The CREMA-D detection rule has no practical threshold, and separate scalar proxy controls do not identify a shortcut in the feature-mixture target.Source: §6 · p. 9
  • In Study III, five-feature-zero has lower raster RMSE than aligned audio, while aligned audio has lower segment-drift error.The whole-frame target and four-video test set support descriptive proxy findings, not rendered-video quality, natural-mixture performance, or speaker attribution.Source: §6 · pp. 9–10
  • Human perceptual evaluation remains future work.Source: §6 · p. 10

Reproducibility Statement

The reproducibility statement describes the protocol materials and the boundary on replay.

Reproducibility Statement

AI Use Statement

The statement identifies AI assistance with literature search, manuscript review and editing, and preparation of final packages, while assigning responsibility for the final text and citations to the authors.

AI Use Statement

A SpeakerVid test contrasts by condition and source video

On the SpeakerVid proxy, five-feature-zero has lower whole-frame raster error than aligned audio, while aligned audio has lower flattened-window segment-drift error. The comparisons are descriptive on a four-video test subset and do not estimate uncertainty over new videos.

A SpeakerVid test contrasts by condition and source video

  • Five-feature-zero has lower raster-reconstruction RMSE, while aligned audio has lower flattened-window segment-drift error across all four conditions.The drift proxy does not measure continuous-video quality; the platform ranges are descriptive, not confidence intervals.Source: A SpeakerVid test contrasts by… · p. 12, Table 8 · p. 12
  • Zeroing the five audio summaries lowers raster-reconstruction error in every condition, while removing the condition branch also lowers mean error.Dummy conditions raise error for every seed; shuffled conditions raise mean error slightly, with seed-level direction varying.Source: A SpeakerVid test contrasts by… · p. 12, Table 9 · p. 12
  • Every held-out video-by-condition cell has a negative five-feature-zero difference on all four platforms, with all five paired seed differences negative.The 108 unique windows come from four videos; the stronger mixed-condition result for Unit 2 is descriptive within-source evidence.Source: A SpeakerVid test contrasts by… · pp. 12–13, Table 10 · p. 13
  • Repeated windows, conditions, seeds, and platforms do not increase the four-video independent sample, so the proxy result does not establish performance on new videos.Output sensitivity to audio or condition inputs does not itself establish a benefit on this compact visual target.Source: A SpeakerVid test contrasts by… · p. 13

B RAVDESS raster confirmation at the registered target

The registered RAVDESS raster test scores 240 fixed pairs at the 2.0-second target, but its detection rule fails because the actor-audio replacement interval includes zero. Separate same-target fits favor constant audio across four matched seeds, with results limited to raster reconstruction on opened confirmation actors.

B RAVDESS raster confirmation at the registered target

  • The protocol scores a 2.0-second, 64 × 64 target on 240 fixed pairs from 16 development, four validation, and four confirmation actors.Inputs exclude emotion, intensity, statement, and actor identifiers; replacement audio changes actor and utterance together.Source: B RAVDESS raster confirmation at the… · p. 13
  • The registered detection rule fails: zero-audio and identity-raster replacement contrasts are positive, but actor-audio replacement is −0.00045 MAE with interval [−0.00129, 0.00039].The actor-audio swap also changes utterance, so it does not isolate a speaker effect.Source: B RAVDESS raster confirmation at the… · p. 13
  • Exploratory metadata-matched cross-actor audio has a −0.00102 MAE delta, but its actor-cluster interval [−0.00491, 0.00141] crosses zero.Four actor clusters do not support population inference, and exploratory controls do not change the registered decision.Source: B RAVDESS raster confirmation at the… · p. 13
  • Constant-audio fits have lower MAE than aligned-audio fits on the same 240 confirmation targets in all four matched seeds.The comparator was designed after confirmation actors had been opened; the runs do not establish perceptual video quality or a prospectively registered benefit.Source: B RAVDESS raster confirmation at the… · p. 14, Table 11 · p. 14
  • With an 80-epoch cap, all four fresh constant-audio fits again have lower MAE, with differences from −0.00307 to −0.00070.The fresh trajectories differ before epoch 40, so their differences cannot be assigned solely to the longer budget.Source: B RAVDESS raster confirmation at the… · p. 14, Table 11 · p. 14
  • A frozen check found a −0.00060 MAE change after continuation for seed 1704, but this retrospective fit does not establish an actor-population effect.All reported RAVDESS results use the 2.0-second target and remain limited to held-out actor raster reconstruction.Source: B RAVDESS raster confirmation at the… · p. 14

C Development checks that stopped visible-speaker routes

Visible-speaker routes stopped before renderer fitting or confirmation when development checks did not meet their endpoint criteria. These checks therefore do not provide negative held-out renderer results or a compatible external evaluator for a visible-speaker statement.

C Development checks that stopped visible-speaker routes

  • The AMI and HDTF source routes stopped before renderer fitting or confirmation after timing checks failed or produced too few positives.These are endpoint-eligibility findings, not negative held-out renderer results.Source: C Development checks that stopped… · p. 15
  • On opened GRID clips, evaluators disagreed across clips, and the source-local predictor passed its directional check for four of six validation talkers, below the preregistered five.The talker-bootstrap interval [−0.01798, 0.06214] crossed zero; confirmation talkers remained unopened.Source: C Development checks that stopped… · p. 15
  • The development checks do not supply a compatible external evaluator for a visible-speaker statement.The paper consequently reports controlled raster and visual-proxy outcomes instead of a positive visible-speaker result.Source: C Development checks that stopped… · pp. 14–15
Loading 2610.15327v1…