PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoASMR-Bench: Can AI-Generated ASMR Videos Fool VLMs and Humans?

Jiaqi Wang, Weijia Wu, Yi Zhan, Rui Zhao, Ming Hu, James Cheng, Wei Liu, Philip Torr, Kevin Qinghong Lin

arXiv:2512.13281v4cs.CV

TL;DR

Video benchmarks centered on broad semantic and physical criteria have limited power to distinguish increasingly realistic AI-generated videos. VideoASMR-Bench evaluates fine-grained audio-visual perception through an adversarial VLM–VGM benchmark, finding that VLMs struggle to detect generated ASMR videos while humans identify them more readily.

  • Problem

    Existing video evaluation suites focus on semantic and physical perspectives, which offer limited discriminative power as generated videos become increasingly realistic.

    Source: §1 · p. 1

  • Method

    VideoASMR-Bench pairs 1,500 social-media ASMR clips with 2,235 synthetic videos from nine VGMs and evaluates generation and detection in an adversarial framework.

    Source: §1 · p. 2, §3.4 · p. 6

  • Results

    VLMs remain unreliable at detecting generated ASMR videos, scoring 45 versus humans’ 70, while current VGMs produce videos that are difficult for VLMs to distinguish.

    Source: §1 · p. 2, §8 · p. 10

  • Takeaways & Limitations

    ASMR provides a testbed for evaluating fine-grained physical fidelity and precise audio-visual synchronization in video models.

    Source: §1 · p. 2, D Limitation · p. 17

    The benchmark centers on ASMR as its evaluation domain, so its evidence concerns that domain.

    Source: D Limitation · p. 17

Abstract

from arXiv · show

With AI-generated videos increasingly indistinguishable from reality, current benchmarks primarily focus on broad semantic alignment and basic physical consistency, offering limited discriminative power for evaluating them. To address this, we introduce VideoASMR-Bench, a benchmark based on Autonomous Sensory Meridian Response (ASMR) videos that emphasizes fine-grained audio-visual perception and sensory immersion. This benchmark aims to answer two key questions: (i) Are today's video understanding models (VLMs) sensitive enough to detect AI-generated ASMR videos by recognizing minor visual, physical, or auditory artifacts? (ii) Can today's video generation models (VGMs) produce convincing ASMR videos with immersive experiences? This benchmark comprises a diverse set of 1,500 high-quality real ASMR videos curated from social media, alongside 2,235 synthetic counterparts generated by nine VGMs. Additionally, we open-source an extensible suite of prompts and reference images, enabling the benchmark to scale dynamically with future video models. Moreover, we design an automatic understanding-generation evaluation framework between VGMs and VLMs, where VGMs aim to produce realistic fake videos to fool the VLMs, while the VLMs seek to detect them, forming an adversarial game between the two parties. Our evaluation on VideoASMR-Bench reveals that even state-of-the-art VLMs, such as Gemini-3-Pro, fail to reliably detect AI-generated ASMR videos. Meanwhile, current frontier video generation models can produce ASMR videos that are difficult for VLMs to distinguish from real ones, while humans can still identify them relatively easily.

1 Introduction

VideoASMR-Bench uses ASMR to test fine-grained audio-visual perception and sensory immersion in video understanding and generation. Its evaluation finds that VLMs struggle to detect synthetic ASMR, while generated videos can fool VLMs even though humans identify them more readily.

1 Introduction

  • VideoASMR-Bench targets fine-grained audio-visual perception, precise temporal-rhythmic alignment, and sensory-affective impact as challenges for video generation and understanding models.The paper presents this as the first systematic study of both tasks in the ASMR domain.Source: §1 · p. 2
  • The benchmark pairs 1,500 social-media ASMR clips with 2,235 synthetic videos from nine VGMs, using prompt and image suites across four settings.It distinguishes easy, short single-step actions in uniform scenes from hard, longer multi-step activities involving diverse materials and complex environments; the suites support conditional synthesis of additional fakes.Source: §1 · pp. 2–3
  • Its adversarial evaluation has VGMs generate fakes to fool VLMs while VLMs try to detect them.This frames understanding and generation as opposing objectives in the benchmark.Source: §1 · p. 2
  • Gemini-3-pro achieves a detection score of 45 versus 70 for humans, and only 12.54% of Veo3.1-generated videos are correctly identified as fake.The evaluation also finds that current VLMs remain unreliable while generated ASMR videos can be difficult for VLMs to distinguish from real ones.Source: §1 · p. 2
  • The work presents ASMR as a testbed for studying immersive sensory experience in advanced video understanding and generation.The benchmark’s social-media video suite and specialized prompt and image suites support adding synthetic videos through conditional synthesis.Source: §1 · p. 3

2 Related Work

Related video benchmarks often assess visual quality, broad semantics, or coarse physical consistency, while giving limited attention to fine-grained audio-visual realism. VideoASMR-Bench is positioned as a joint test of generation and perception centered on ASMR.

2 Related Work

  • Video-generation benchmarks commonly assess visual quality, temporal consistency, and text-video alignment, while physical benchmarks emphasize gross violations rather than fine-grained material interactions.Video-understanding benchmarks often focus on high-level captioning or question answering and treat audio as secondary or omit it.Source: §2 · p. 3
  • Existing AI-video detection benchmarks predominantly use visual cues, while joint audio-video synthesis evaluations often settle for coarse synchronization.Omni-modal models integrate audio but lack dedicated protocols for synthetic-content detection.Source: §2 · p. 3
  • VideoASMR-Bench uses strict audio-visual alignment and sensory immersion to distinguish real from fake, combining fine-grained physical realism with adversarial evaluation.The paper characterizes it as the first benchmark to make these criteria central.Source: §2 · p. 3

3 VideoASMR-Bench

VideoASMR-Bench combines real ASMR videos with prompt and image suites for generating synthetic clips, then evaluates generators and video-understanding models in an adversarial framework. It is designed to cover varied actions and clip complexity while remaining extensible to new generation models.

3.1 Collect ASMR Suite Families

  • The ASMR Video, Prompt, and Image Suites provide real clips and conditioning material for synthesizing additional fake videos.The image suite uses matched reference images, and the prompt suite describes scenes, subjects, actions, temporal dynamics, and sounds.Source: §3.1 · p. 4

3.2 Synthetic Fake ASMR Videos

  • The synthetic clips are generated using six VGMs in the default text-image-to-video setting and three additional settings: text-to-image-to-video, text-only, and image-only.The reported generation processes yield 894 videos in the default setting and 1,192 across the additional settings.Source: §3.2 · p. 5

3.3 Quality Verification

  • The benchmark contains 2,235 real–fake videos across easy and hard subsets, nine VGMs, and four generation settings.Its manually verified prompt and image suites are intended to support continued inclusion of new generation models while preserving evaluation consistency.Source: §3.3 · pp. 5–6
  • Actions, backgrounds, and objects vary across the dataset; cutting and crushing account for 31% of actions, while spreading and mixing account for 23%.Easy videos are typically 3–5 seconds, whereas hard videos can exceed 20 seconds.Source: §3.3 · p. 6

3.4 Adversarial Understanding-Generation Evaluation

  • The benchmark formalizes an adversarial game in which VGMs generate fake videos to fool VLMs, while VLMs try to detect them.VLMs are evaluated on balanced real/fake inputs with accuracy aggregated across VGMs; model responses also require step-by-step reasoning.Source: §3.4 · p. 6

4 Evaluation for ASMR Video Understanding

The evaluation tests video-understanding models on real ASMR videos and generated fakes, using video and audio inputs where available. Models generally trail human performance, with different models leading on accuracy and F1.

4 Evaluation for ASMR Video Understanding

  • The evaluation uses real ASMR videos and generated fakes, with eight uniformly sampled interleaved frames as input; text-only models receive prompts only.Fake videos use the ImgText2Vid setting with the ASMR prompt and image suites where applicable.Source: §4 · p. 7
  • Most VLMs achieve below 70 accuracy and below 45 F1, compared with human scores of 87 accuracy and 70 F1; Gemini-3-pro leads accuracy, while GLM-4.5V leads average F1.Proprietary models generally have stronger accuracy than most open-source models, but their F1 scores remain lower than or comparable to open-source models.Source: §4 · p. 7

5 Evaluation for ASMR Video Generation

The generation evaluation compares models and settings using the benchmark’s ASMR prompts and images. VLMs identify only a small share of the strongest models’ generated clips as fake.

5 Evaluation for ASMR Video Generation

  • The default setting uses text and images from the ASMR suites, with additional text-to-image-to-video, text-only, and image-only settings.The image-only setting uses the minimal prompt “Generate the video as real as possible.”Source: §5 · pp. 7–8
  • 12.54% of Veo3.1-Fast videos are identified as fake, the lowest reported detection rate; HunyuanVideo-I2V reaches 14.77% and Wan2.2-A14B 16.10%.The table reports Fakeness scores for generation models under different settings, where lower scores indicate better performance.Source: Table 3 · p. 8, §5 · p. 8

6 Observations and Insights

The observations identify audio alignment, watermarks, and a real-labeling bias as factors associated with VLM detection performance. Prompting and comparison format also affect reported realism scores and detection accuracy.

6 Observations and Insights

  • Adding audio improves detection by an average of 5 points on the two most realistic fake sets from Veo3.1 and Sora2.The authors attribute the improvement to inadequate audio–visual alignment in current generators; Sora2 often produces speech, while Veo3.1-fast produces more action-aligned audio without speech.Source: §6 · pp. 8–9
  • After Sora2 watermark removal, several strong VLMs’ performance drops sharply, with an average decrease of about 30 points.The authors report that leading VLMs such as Gemini-2.5 and GPT-5 often rely on the watermark rather than intrinsic authenticity cues.Source: §6 · p. 9
  • Across VLMs, predictions show a consistent bias toward labeling videos real rather than fake.The trend also appears on the easy split.Source: §6 · p. 9
  • The ASMR prompt suite consistently improves generated-video realism scores over a minimal realism-only instruction.The reported gains are especially notable for GPT-4-turbo and Gemini-2.5-pro.Source: §6 · p. 9
  • Pair preference achieves higher realism scores than direct judgment, while pair and pair-shuffle settings have little impact; explicit reasoning also enhances detection accuracy.The comparison spans models and prompt settings.Source: §6 · p. 9

7 Qualitative Examples

Qualitative cases show that detection can depend on superficial watermarks or audio input, while realistic Veo3.1-Fast clips can fool VLMs.

7 Qualitative Examples

  • Gemini-2.5-Pro labels a Sora2 clip fake when its watermark is visible but real after watermark removal, indicating reliance on a detection shortcut.The passage contrasts watermark-based classification with distinguishing clips through perceptual cues.Source: §7 · p. 9
  • Gemini-2.5-Flash initially classifies Sora2 ASMR clips as real but detects them as fake when audio is included.The passage attributes this change to Sora2’s lack of convincing audio-visual coherence from a human perceptual perspective.Source: §7 · p. 9
  • The passage reports that lighting consistency led GPT-5 and Gemini-2.5-Pro to mistakenly classify the clip as real, and says additional qualitative examples appear in App.A&B.Source: §7 · p. 10

8 Conclusion

VideoASMR-Bench is presented as a testbed for fine-grained audiovisual realism, and the examples show both VLM detection difficulty and generation artifacts.

8 Conclusion

  • Top-tier VLMs remain unreliable at detecting generated ASMR, while current VGMs can fool them but still fail to deceive human annotators.The conclusion characterizes the benchmark as a 1,500-video social-media collection paired with a specialized prompt and image suite for conditional generation.Source: §8 · p. 10

Appendix

  • Figure 9 pairs real ASMR clips with Sora2-generated counterparts across tomato, vegetable, and pitaya cutting scenarios.The generated examples include a tomato retaining its shape after slicing and a knife cutting into a hand, while VLMs struggle to identify such artifacts.Source: Figure 9 · p. 13, §A.1 · p. 13

A.1 Easy Split Examples for Sora2

  • Easy-split Sora2 clips contain conspicuous physical artifacts, yet video understanding models struggle to detect even these failures reliably.The examples include implausible cutting outcomes in food-cutting sequences.Source: §A.1 · p. 13

A.2 Hard Split Examples for Sora2 and Veo3.1

Hard-split examples contrast plausible Veo3.1-Fast outputs with Sora2 deviations, and the audio comparisons show that adding audio can change VLM judgments.

A.2 Hard Split Examples for Sora2 and Veo3.1

  • Veo3.1-Fast outputs are visually plausible and difficult to distinguish from real footage, while Sora2 often deviates from the prompts and images.The hard split includes diverse, multi-step scenarios and natural interaction sounds that demand richer visual and auditory coherence.Source: §A.2 · pp. 14–15, Figure 11 · p. 15, Figure 12 · p. 16
  • Sora2 audio is dominated by human voices rather than expected object-interaction sounds, suggesting possible overfitting to human-centric audio patterns in training data.The hard examples include subtle sounds such as water being squeezed and foam dissolving.Source: §A.2 · pp. 14–16

B Reasoning Traces Examples.

B.1 Comparison with and without Audio for Veo3.1

  • Veo3.1-Fast is detected as generated with audio but classified as real without audio.Source: §B.1 · p. 16

B.2 Comparison with and without Audio for Sora2

  • Sora2 is detected as generated with audio but classified as real without audio.Source: §B.2 · p. 16

D Limitation

The authors frame VideoASMR-Bench as a benchmark and evaluation protocol, not a methodological advance, and acknowledge potential subjectivity in human annotation.

D Limitation

  • The paper’s contribution is a benchmark and evaluation protocol rather than a methodological advance.Source: D Limitation · p. 17
  • Human annotation may introduce subjective bias, while the framework uses diverse VLMs and VGMs in mutual supervision for assessment.Source: D Limitation · p. 17
  • The protocol includes preference-based evaluation and in-depth CoT analysis alongside standard judgment tasks.The authors say models explicitly reason about audio-visual incoherence.Source: D Limitation · p. 17

E Evaluation metric

The evaluation formalizes an adversarial game in which generators create fake ASMR videos from a reference image and prompt, while understanding models classify videos as real or fake. It defines complementary metrics and prompt variants for assessing detection and generated-video realism.

E Evaluation metric

  • The adversarial protocol conditions a generator on an ASMR reference image and prompt, then asks an understanding model to classify videos as real or fake.Predictions are parsed from the required answer tags; only valid predictions are counted.Source: E Evaluation metric · pp. 17–18

E.1 Understanding Model Detection Accuracy.

  • Detection accuracy aggregates an understanding model’s performance across generators, while macro-averaged per-generator F1 better captures missed detections and false alarms.The evaluation balances real and fake videos; an always-real classifier would achieve 50% accuracy but have no discriminative power.Source: §E.1 · pp. 19–20

E.2 Generation Model Realism Scores.

  • A generation model’s fakeness score measures how often its generated videos are detected as fake across understanding models; lower scores indicate more realistic outputs.The framework interprets higher detection accuracy as stronger understanding-model performance and lower fakeness as greater generator realism.Source: §E.2 · p. 20

F Video generation settings

  • OpenSora V2, Wan 2.2, other open-source generators, Veo3.1, and Sora2 use their default or specified implementations and generation settings.The tested Wan models are T2V-A14B, I2V-A14B, and TI2V-5B; Sora2 is also tested with image-to-video settings and with watermarks removed for an ablation.Source: F Video generation settings · p. 20, Table 4 · p. 21, Table 5 · p. 21

G.1 Real–Fake Detection Prompt.

  • The default detection prompt requires explicit reasoning before a binary real-or-fake answer, while a separate judgment prompt requests only the binary decision.The reasoning prompt is used for Tables 3 and 13; the direct-judgment prompt is used for Table 4.Source: §G.1 · p. 22, §G.2 · p. 22

G.3 Preference Comparison Prompt with Reasoning.

  • Preference-comparison prompts ask models to select which of two videos is real, either after brief reasoning or with a direct answer.The direct-answer variant uses “1” and “2” for the first and second videos, respectively.Source: §G.3 · p. 22, §G.4 · p. 22

G.5 ASMR Prompt Suite Generation.

  • The ASMR Prompt Suite is generated by prompting for one coherent paragraph from eight uniformly sampled frames, integrating setting, objects, actions, temporal dynamics, and implied sounds.Easy prompts use Sora2’s storyboard module; hard-level prompts are generated with Gemini-2.5-Pro and manually inspected.Source: §G.5 · p. 23

H.1 Detection Results

Across the tested VLMs, detection performance remains below human performance, with proprietary models generally outperforming most open-source models. Results also report intra-family detection differences and evaluations of reasoning, pairing, preference, and other conditions.

H.1 Detection Results

  • 76.27 average accuracy across VGM-generated fake videos is about 13 points below the human score of 89.11, despite Gemini-3-pro leading the VLMs.Most open-source models perform poorly, and the table compares 11 VLMs with human and random baselines.Source: §H.1 · p. 23, Table 13 · p. 25
  • Reasoning improves Opensora-v2 accuracy and makes Wan2.2-A14B scores identical across Pair and Pair shuffle modes, while pairing type can produce volatile scores without reasoning.The pairing volatility is prominent in simple preference tasks.Source: Table 10 · p. 24, Table 11 · p. 24
  • VLMs show a potential intra-family bias, with models tending to score videos from corresponding proprietary generators unusually low, such as Gemini on Veo outputs.The authors hypothesize this may stem from VLMs serving as reward models or evaluators during generator optimization.Source: §H.1 · p. 24
  • Preference evaluation is reported as easier than judgment in the Qwen2.5-VL-7B ablation, alongside comparisons of reasoning and prompt conditions.Table 12 covers thinking versus no thinking and generation with or without the ASMR prompt suite.Source: Table 12 · p. 25
  • The ablation reports that VLM performance drops after Sora watermark removal, adding audio to visual inputs generally improves reality-detection accuracy, and using more input frames degrades VLM detection performance.Source: Figure 16 · p. 25

H.2 Real-Fake Evaluation Results

  • Tables 14–23 report hard-level real-versus-fake evaluation results and prediction statistics for individual understanding models.The listed models include Qwen, GPT, Gemini, and GLM variants.Source: §H.2 · p. 25, Table 14 · p. 26, Table 15 · p. 26

Related papers

Loading 2512.13281v4…