Source-linked AI summary
TAU: A Benchmark for Cultural Sound Understanding Beyond Semantics
Yi-Cheng Lin, Yu-Hua Chen, Jia-Kai Dong, Yueh-Hsuan Huang, Szu-Chi Chen, Yu-Chen Chen, Chih-Yao Chen, Yu-Jung Lin, Yu-Ling Chen, Zih-Yu Chen, I-Ning Tsai, Hsiu-Hsuan Wang, Ho-Lam Chung, Ke-Han Lu, Hung-yi Lee
TL;DR
Audio benchmarks emphasize speech semantics or globally sourced sounds, leaving localized, non-semantic audio underexamined. TAU is a Taiwan-focused soundmark benchmark built with acoustic-only questions and transcript-shortcut filtering. Local humans outperform the evaluated systems, and culturally specific prompting yields selective rather than consistent gains.
Problem
Audio evaluation emphasizes speech semantics or globally sourced environmental sounds, leaving open whether models recognize localized, non-semantic audio that communities identify from sound alone.
Source: §1 · p. 1Method
TAU evaluates Taiwanese soundmarks with multiple-choice questions grounded in acoustic signatures and filters out items answerable through transcript-only shortcuts.
Source: §1 · p. 1, §3.1 · p. 2, §3.3.4 · p. 3Results
Humans score 84.0% on Single-hop and 83.3% on Multi-hop questions, ahead of all evaluated systems; Gemini 2.5 Pro leads the LALMs at 72.4% / 73.9%.
Source: §4.2 · p. 4Takeaways & Limitations
The findings support culturally informed data, tasks, and metrics for assessing real-world robustness and region-dependent failure modes.
Source: §6 · p. 4TAU is centered on Taiwan-specific soundmarks, and its finite collection may overrepresent urban settings while undersampling rural contexts and some cultural practices.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Large audio-language models are advancing rapidly, yet most evaluations emphasize speech or globally sourced sounds, overlooking culturally distinctive cues. This gap raises a critical question: can current models generalize to localized, non-semantic audio that communities instantly recognize but outsiders do not? To address this, we present TAU (Taiwan Audio Understanding), a benchmark of everyday Taiwanese "soundmarks." TAU is built through a pipeline combining curated sources, human editing, and LLM-assisted question generation, producing 702 clips and 1,794 multiple-choice items that cannot be solved by transcripts alone. Experiments show that state-of-the-art LALMs, including Gemini 2.5 and Qwen2-Audio, perform far below local humans. TAU demonstrates the need for localized benchmarks to reveal cultural blind spots, guide more equitable multimodal evaluation, and ensure models serve communities beyond the global mainstream.
1. INTRODUCTION
Existing audio evaluation includes environmental sound, but current benchmarks emphasize speech semantics or generic, globally sourced sounds rather than localized cultural cues. TAU addresses this gap with Taiwanese non-speech soundmarks and questions designed to require acoustic or cultural understanding.
1. INTRODUCTION
- Current audio benchmarks often test speech semantics or globally common environmental categories, leaving open whether models recognize localized, non-semantic audio familiar to communities but rare to outsiders.Examples of globally common sounds include sirens, dog barks, doorbells, rainfall, and engine noise.Source: §1 · p. 1
- Ignoring soundmarks can distort benchmark validity by overestimating real-world competence and understating generalization challenges across geographic and cultural contexts.Soundmark recognition depends on cultural exposure rather than linguistic comprehension.Source: §1 · p. 1
- TAU curates everyday Taiwanese non-speech sounds and uses multiple-choice questions that cannot be answered by semantic reasoning alone, targeting timbre, rhythm, and iconic acoustic patterns.The paper presents Taiwan as a case study for a pathway toward localized audio benchmarks.Source: §1 · p. 1
2. RELATED WORKS
Prior audio-language work spans chat-based understanding, general-purpose multi-task suites, and skill-specific stress tests, while culture-focused benchmarks largely target text or images. TAU targets the remaining gap: recognizing locale-specific, non-semantic acoustic cues.
2. RELATED WORKS
- Existing audio-language models span general-purpose perception, instruction following, long-context reasoning, and lightweight audio-language designs [3] [4] [15] [16].The related work describes Qwen2-Audio [4], DeSTA2.5-Audio [3], Audio-Reasoner [15], and Gemma-3n [16].Source: §2 · p. 1
- Chat-based audio benchmarks assess dialogue and comprehension, while multi-task suites cover speech, non-speech, and paralinguistics, and domain-focused tests target specific skills [19] [24].Examples include SD-Eval, AudioBench, Audio Entailment, and MuChoMusic.Source: §2 · pp. 1–2
- Culture-focused resources evaluate local knowledge across regions or Taiwanese culture, but the cited benchmarks are text- or image-centric [30].BLEnD, CulturalBench, ThaiCLI, TaiwanVQA, and VisTW [30] are examples.Source: §2 · p. 2
3.1. Design Principles
TAU’s design principles prioritize locally identifiable soundmarks, acoustic evidence independent of lexical content, and broad but accessible coverage. Together, they define which sounds and questions belong in the benchmark.
3.1. Design Principles
- TAU prioritizes everyday soundmarks that Taiwanese listeners recognize from exposure and non-locals are unlikely to identify without prior exposure.Examples include public signals, transportation cues, and consumer-device prompts.Source: §3.1 · p. 2
- Items must be answerable from acoustic signatures such as timbre, rhythm, envelope, or characteristic patterns rather than lexical content.Speech fragments are trimmed or counter-balanced so transcripts alone cannot reveal the answer.Source: §3.1 · p. 2
- The benchmark seeks breadth across venues, devices, and activities while avoiding phenomena so rare or specialized that typical residents would not encounter them.The aim is coverage of culturally salient sounds accessible without specialized knowledge.Source: §3.1 · p. 2
3.2. Roles
The annotation workflow divides sound selection, quality control, item editing, and human validation among editors, checkers, and reviewers.
3.2. Roles
- Ten native Taiwanese annotators from different regions and gender-balanced backgrounds serve as editors, checkers, and reviewers across sound nomination, quality control, item refinement, and human validation.Checkers verify timing, audibility, and semantic independence; reviewers confirm items are solvable and appropriately challenging.Source: §3.2 · p. 2
3.3. Workflow
TAU’s workflow combines culturally informed sound selection, audio review, human-edited question generation, and transcript-only filtering. The resulting items are designed to test acoustic or cultural understanding rather than lexical shortcuts.
3.3. Workflow
- The workflow links soundmark selection and clip review to human-edited MCQ creation and transcript-only filtering.The collection workflow is summarized in Fig. 1.Source: §3.3 · p. 2, §3.3.1 · p. 2, §3.3.2 · p. 3
3.3.1. Concept collection
- Editors compiled 550 Taiwan-specific soundmarks, prioritizing locally recognizable cues while avoiding rare or expert-only cases.Checkers assessed the candidates against the design principles and assigned provisional categories and descriptors.Source: §3.3.1 · p. 2
3.3.2. Audio Collection
- Approved targets came from Creative Commons repositories and team recordings; clips were checked for target audibility, continuity, and semantic leakage.The collection attached up to three variants per target that differed by location, background conditions, or device/version.Source: §3.3.2 · pp. 2–3
3.3.3. Question Generation
- Gemini 2.5 Flash drafted four-option questions from minimal descriptors, then editors refined wording and culturally plausible distractors.Items probe different facets; Single-hop needs one acoustic cue, while Multi-hop combines acoustic evidence with background knowledge.Source: §3.3.3 · p. 3
3.3.4. Question filtering
- A transcript-only LLM attempted each question, and items were discarded when its success rate significantly exceeded 25% chance at p < 0.05.The test used five sampled responses per item and a one-tailed t-test.Source: §3.3.4 · p. 3
3.4. Dataset Statistic
TAU contains 702 clips across ten culturally distinctive categories and 1,794 multiple-choice items. Its category mix reflects everyday Taiwanese sound frequency, while recording variants broaden contextual coverage.
3.4. Dataset Statistic
- 702 clips cover 10 culturally distinctive categories, with an intentionally imbalanced mix reflecting everyday sound frequency.The collection includes frequent urban cues as well as less common but socially important sounds.Source: §3.4 · p. 3
- 1,794 multiple-choice items accompany the clips; median duration is 9.43 seconds and the design caps clips at 30 seconds.The authors describe this duration design as balancing realism with usability.Source: §3.4 · p. 3
- Each soundmark has an average of 2.1 recording variants differing by location, time, or background conditions.The authors state that this variation increases robustness and reduces overfitting to specific contexts.Source: §3.4 · p. 3
4. EXPERIMENTS
On TAU, humans outperform the evaluated audio-language models, and multi-hop questions are harder for weaker systems. Culturally grounded prompting produces model-specific shifts rather than consistent gains, and does not close the human gap.
4.1. Evaluation protocol
- The evaluation compares default and culturally grounded prompts, alongside random, ASR+LLM, LLM-only, and human baselines.Nine annotators completed the benchmark, with each question evaluated by two independent annotators.Source: §4.1 · p. 4
4.2. Performance with Default System Prompt
- Humans score 84.0% / 83.3% on Single-hop / Multi-hop, ahead of Gemini 2.5 Pro at 72.4% / 73.9%.The reported figures use each model’s default prompt.Source: §4.2 · p. 4
- LLaMA-3.1 ASR+LLM scores 34.9% / 34.1%, while text-only LLM baselines stay around 35–41%.Multi-hop accuracy also lags Single-hop across weaker systems.Source: §4.2 · p. 4
4.3. Performance with Local Specific Prompt
- Gemini 2.5 Pro falls from 72.4% / 73.9% to 70.6% / 71.8% under the culturally grounded prompt, while Gemma-3n variants gain on Multi-hop items.Gemma-3n variants rise from about 29% to 33–34%; prompting alone is insufficient to close the model–human gap.Source: §4.3 · p. 4
- DeSTA2.5-Audio and Qwen2.5-Omni-7B show mixed changes, so cultural priming does not consistently improve accuracy.The authors suggest exploring explicit cultural grounding in training rather than relying solely on prompt engineering.Source: §4.3 · p. 4
5. LIMITATIONS
TAU is limited to Taiwan-specific soundmarks, a finite and potentially uneven sample, and sounds that may change over time.
5. LIMITATIONS
- TAU’s Taiwan-specific soundmarks limit what its results establish about competence in other locales.Strong performance does not guarantee competence elsewhere, while weak performance may reflect cultural unfamiliarity rather than general auditory shortcomings.Source: §5 · p. 4
- The finite sound pool may overrepresent urban scenes, undersample rural contexts and cultural practices, and leave device-related covariate shifts uncontrolled.The pool also covers a limited range of venues, devices, and times of day.Source: §5 · p. 4
- Soundmarks can change over time, so evaluated models may face distribution shifts as physical environments evolve.The authors propose dated benchmark snapshots and lightweight incremental updates rather than a single frozen dataset.Source: §5 · p. 4
6. CONCLUSION
The conclusion presents TAU as a reproducible pathway for localized, culturally grounded audio evaluation. Current audio–language models fall short of human performance, and culturally specific prompting offers at best selective gains.
6. CONCLUSION
- Current audio–language models fall well short of human performance on localized items, while culturally specific prompts provide at best selective gains.The authors conclude that prompt engineering alone is insufficient.Source: §6 · p. 4
- TAU’s five-stage pipeline combines concept curation, licensed sourcing, quality control, LLM-assisted item generation, and leakage filtering as an adaptable template.The authors present it as a reproducible pathway for other regions and communities.Source: §6 · p. 4
- The findings underscore the need for culturally informed data, tasks, and metrics to assess real-world robustness and support equitable multimodal deployment.The authors also identify mitigating region-dependent failure modes as a motivation.Source: §6 · p. 4