PaperlayerView exact revision on arXiv

Source-linked AI summary

ModalFidelity: Routing Modalities for Deepfake Detection on a Budget

Oguzhan Baser, Kaan Kale, Sriram Vishwanath, Sandeep Chinchali

arXiv:2609.38246v1cs.CRcs.AI

TL;DR

Targeted deepfakes can occupy only a small, unknown part of a stream, while existing detectors either miss modalities or analyze every window. ModalFidelity previews windows and routes which stream to acquire under a hard compute budget. On AV-Deepfake1M, it is more accurate than gating after the detectors at 15.9× less compute and retains over 96% of clairvoyant-oracle accuracy.

  • Problem

    Targeted deepfakes occupy a small, unknown share of streams, while existing detectors either miss modalities and temporal localization or process every window.

    Source: §1 · p. 1

  • Method

    ModalFidelity uses a cheap preview to choose window by window which modality to acquire before forensic detectors run, subject to an explicit compute budget.

    Source: §1 · p. 2

  • Results

    ModalFidelity is more accurate than gating after the detectors at 15.9× less compute and retains over 96% of a clairvoyant oracle’s accuracy.

    Source: §1 · p. 2

  • Takeaways & Limitations

    For targeted forgeries with sparse, asymmetric evidence, uniform effort is the wrong default.

    Source: §1 · p. 2

    The router commits each window’s action before the rest of the stream exists, without knowing how many edits there are or where they fall.

    Source: §2 · p. 2

Abstract

from arXiv · show

Deepfakes no longer need to fake a whole video. Generators that read the transcript now alter only the few seconds in which a video's meaning turns, so a forgery hides in a small, unknown fraction of the video. Yet detectors still read every one-second window of both the audio and image streams, spending nearly all of their compute where nothing was altered. We observe that deciding where to look is far cheaper than looking. We present ModalFidelity, a lightweight router that previews each window and decides, before any forensic detector runs, which stream is worth reading, under a hard compute budget it can never exceed. On AV-Deepfake1M, reading at most a fifth of the windows, it is more accurate than gating after the detectors at 15.9x less compute, and retains over 96% of the accuracy of an oracle that knows where every forgery lies.

1. INTRODUCTION

Targeted, semantically chosen edits make a small, unknown share of multimodal streams the detection challenge, while exhaustive detectors spend on untouched windows. ModalFidelity routes acquisition before forensic detectors under an explicit compute budget, with reported gains over post-detector gating and near-oracle accuracy.

1. INTRODUCTION

  • Transcript-guided generators can alter only semantically decisive seconds, making edits short and harder to find while detectors still spend effort on untouched windows.The paper describes generators locating words such as a name, dosage, or negation before altering the corresponding span.Source: §1 · p. 1
  • Existing detectors either miss the unread modality and temporal localization or process every stream on every window, leaving selective, budgeted localization unresolved.The missing detector would jointly select what to acquire, localize by time and modality, and operate at a verifier-fixed fraction of the cost of reading every window.Source: §1 · pp. 1–2
  • A cheap preview lets ModalFidelity choose which modality to acquire window by window before costly detectors run, under an explicit compute budget.In the B=4 illustration, it saves budget for forged windows and flags all three forged streams.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • ModalFidelity is more accurate than gating after the detectors at 15.9× less compute and retains over 96% of an oracle’s accuracy.The oracle knows where every forgery lies.Source: §1 · p. 2
  • The authors release the router model, its training pipeline, and a benchmark for studying where gating pays off in a detector stack.The benchmark emphasizes gating before rather than after the detectors.Source: §1 · p. 2

2. METHODOLOGY

ModalFidelity previews windows and routes detector calls to selected audio or image streams under a hard budget. It makes sequential decisions using the remaining budget and stream history, then pools acquired detector margins into a clip prediction.

  • The unbudgeted gate reaches 0.819 accuracy, versus 0.666 for audio, 0.635 for image, 0.718 for multimodal detectors, and 0.839 for the oracle.Source: Figure 2 · p. 2
  • Acquired streams are thresholded at fixed detector operating points, and max-pooled margins let a single forged window determine the clip prediction.A declined stream defaults to authentic; if nothing is acquired, the clip prediction is authentic.Source: §2 · p. 2
  • The router commits to stream acquisitions before later windows exist, with a hard call quota and a reward that credits manipulated streams and charges for authentic ones.The quota acts as a ceiling rather than a target; decisions are made without lookahead.Source: §2 · pp. 2–3
  • A cheap preview of subsampled frames and spectrograms feeds an LSTM that receives remaining budget and horizon, then scores four actions: none, audio, image, or both.The preview encoder is frozen; only the LSTM and action head are trained.Source: §2 · pp. 2–3
  • Training first distills exact knapsack actions with DAgger, then uses self-critical policy gradients to adapt to preview-only information, raising accuracy by 1.9 to 6.5 pp across budgets.The teacher supplies best actions at the student's visited states, including states reached through mistakes.Source: §2 · p. 3
  • At ρ=0.10, ModalFidelity reaches 0.655 accuracy, versus at most 0.617 for fixed detectors and 0.543 for blind allocators.Source: Figure 3 · p. 3

3. EXPERIMENTAL SETUP

The evaluation tests ModalFidelity on annotated, short-segment manipulations in AV-Deepfake1M and measures window-level detection accuracy and compute cost.

3. EXPERIMENTAL SETUP

  • AV-Deepfake1M contains over 1M videos of more than 2K subjects, with short audio, image, or joint manipulations annotated by stream and time span.Videos are split by identity; 86.6% of held-out windows are authentic.Source: §3 · p. 3
  • The audio and image streams use frozen W2V2-AASIST and GenD detectors, while AVH-Align provides a frozen multimodal alternative.Baselines share the router’s detectors and budget but differ in how they spend it.Source: §3 · p. 3
  • Detection accuracy measures mean recall on manipulated windows and specificity on authentic windows; cost is budget fraction ρ or, for gate placement, GFLOPs per window.Predictions are evaluated against annotated spans, measuring localization at window resolution.Source: §3 · p. 4

4. RESULTS

Results show that routing before detectors improves accuracy while reducing compute, and that useful acquisition depends on both where edits occur and which stream carries them.

4. RESULTS

  • Audio and image detectors fire on only 70% and 64% of edits in their respective streams, while AVH-Align catches 88% of audio edits but only half of image edits.No fixed detector covers both edit types, supporting window-by-window stream selection.Source: §4 · p. 4
  • At ρ=0.30, the router reaches 0.815 accuracy, within 0.4 pp of the unbudgeted gate’s 0.819 and within 2.6 pp of the oracle at every budget.At ρ=0.50, it reaches 0.844; uniform and random allocators remain between 0.51 and 0.60.Source: §4 · p. 4
  • At ρ=0.20, ModalFidelity reaches 0.771 per-video accuracy with 6.27 units average spend, versus 0.577 for uniform, 0.526 for random, and 0.799 for the oracle.Figure 4 plots per-video cost against accuracy; at clip level, the router reaches 0.928 versus 0.940 for the oracle and at most 0.875 for any fixed detector.Source: §4 · p. 4, Figure 4 · p. 4
  • At ρ=0.20, single-detector baselines trail ModalFidelity by 6.6 to 8.1 pp, showing that stream selection adds accuracy beyond window selection.These baselines retain the router’s windows but always read the same stream.Source: §4 · p. 4
  • At ρ=0.20, input routing costs 258 GFLOPs per window—15.9× fewer than late gates—and reaches 0.767 accuracy against at most 0.732.It also uses 15.4× fewer detector calls; at ρ=0.50 it is 10.6× cheaper and 11 pp more accurate. Table 1 compares gate placements.Source: §4 · p. 4, Table 1 · p. 4

5. CONCLUSION

ModalFidelity routes detector calls by window and stream under a hard budget, producing localized predictions. On AV-Deepfake1M, it approaches oracle accuracy with substantially fewer FLOPs than late MoEs.

5. CONCLUSION

  • ModalFidelity decides window by window which stream to acquire under a hard compute budget and outputs per-window, per-stream predictions.The predictions indicate when a forgery occurred and in which stream, at window resolution.Source: §5 · p. 4
  • On AV-Deepfake1M, the router retains over 96% of a clairvoyant oracle’s accuracy at every budget and needs 15.9× fewer FLOPs than late MoEs..Source: §5 · p. 4
  • The authors expect routing to extend beyond deepfakes where evidence is sparse and costly to read, and identify an adversary learning the policy as future work.Source: §5 · p. 4
Loading 2609.38246v1…