PaperlayerView exact revision on arXiv

Source-linked AI summary

PARSA-Bench: A Comprehensive Persian Audio-Language Model Benchmark

Mohammad Javad Ranjbar Kalahroodi, Mohammad Amini, Parmis Bathayan, Heshaam Faili, Azadeh Shakery

arXiv:2603.14456v1cs.CLcs.SD

TL;DR

Existing benchmarks do not capture Persian’s culturally grounded audio phenomena, including poetry meter, traditional music, and code-switching. The paper introduces PARSA-Bench, a 16-task benchmark with over 8,000 samples, and finds that audio processing is the dominant bottleneck while poetry meter detection remains near random chance.

  • Problem

    Existing audio benchmarks lack systematic evaluation of Persian language, paralinguistic, and culturally grounded audio understanding.

  • Method

    PARSA-Bench evaluates eight large audio-language models across 16 tasks and over 8,000 samples spanning speech, paralinguistic, and Persian cultural audio understanding.

  • Results

    Text-only baselines consistently outperform audio counterparts across dimensions, while all models perform near random chance on Persian poetry meter detection regardless of scale.

  • Takeaways & Limitations

    The benchmark indicates that models struggle chiefly to extract usable information from Persian audio, especially culturally grounded prosodic information.

  • Takeaways & Limitations

    Six tasks use TTS audio, human performance baselines are absent, and extended prompting experiments cover only open-source models.

Abstract

from arXiv · show

Persian poses unique audio understanding challenges through its classical poetry, traditional music, and pervasive code-switching - none captured by existing benchmarks. We introduce PARSA-Bench (Persian Audio Reasoning and Speech Assessment Benchmark), the first benchmark for evaluating large audio-language models on Persian language and culture, comprising 16 tasks and over 8,000 samples across speech understanding, paralinguistic analysis, and cultural audio understanding. Ten tasks are newly introduced, including poetry meter and style detection, traditional Persian music understanding, and code-switching detection. Text-only baselines consistently outperform audio counterparts, suggesting models may not leverage audio-specific information beyond what transcription alone provides. Culturally-grounded tasks expose a qualitatively distinct failure mode: all models perform near random chance on vazn detection regardless of scale, suggesting prosodic perception remains beyond the reach of current models. The dataset is publicly available at https://huggingface.co/datasets/MohammadJRanjbar/PARSA-Bench

1. Introduction

Persian audio presents linguistic and cultural phenomena that existing benchmarks do not capture, motivating PARSA-Bench. The benchmark evaluates whether audio-language models can use audio-specific and culturally grounded information beyond text.

  • Motivation: Persian poetry meter depends on prosodic rhythms absent from transcribed text because standard Persian script omits short vowels.Audio recitation is therefore the primary carrier of metrical information.
  • Motivation: Existing benchmarks cover broad audio evaluation but provide no mechanism for Persian’s poetry, Dastgah music, or pervasive Persian-English code-switching.The paper identifies no dedicated benchmark for these challenges.
  • Motivation: Persian audio understanding requires tasks built from the ground up because limited speech data, culturally specific knowledge, and unsuitable evaluation frameworks compound the difficulty.The authors explicitly reject simply adapting English templates.
  • Contribution: PARSA-Bench covers 16 tasks and over 8,000 samples across speech understanding, paralinguistic analysis, and Persian cultural audio understanding.Figure 2 organizes the benchmark around these three evaluation dimensions.
  • Contribution: Across evaluations, text-only baselines consistently outperform audio counterparts, while poetry meter detection remains near random chance even for the largest models.The results identify audio processing as a bottleneck and expose a distinct difficulty with culturally grounded prosodic understanding.

2. Related Work

Prior work has expanded audio-language modeling and broad audio benchmarks, but Persian and culturally grounded non-English speech remain underrepresented. PARSA-Bench addresses this evaluation gap with dedicated Persian language and culture tasks.

  • Audio-language models: Large Audio-Language Models evolved from foundational transcription and retrieval systems toward unified architectures and stepwise reasoning over complex audio.Recent Large Audio Reasoning Models target reasoning-centric audio inputs.
  • Audio benchmarks: Existing benchmarks span broad audio question answering, hierarchical reasoning, speech skills, music understanding, and instruction-tuned tasks.Examples include MMAU, MMAR, AudioBench, MuChoMusic, MMSU, and Dynamic-SUPERB.
  • Cultural coverage: MMAU-Pro finds that models perform strongest on Western and Chinese music while struggling with Indian, Latin American, and Middle Eastern traditions.This result reveals training-data bias in culturally diverse music evaluation.
  • Cultural coverage: No existing benchmark systematically evaluates non-English speech understanding, paralinguistic analysis, and culturally grounded Persian audio phenomena.The related-work discussion identifies linguistic cultural evaluation as especially underexplored.

3. PARSA-Bench Construction

PARSA-Bench is a unified Persian audio benchmark designed to support broad, fair evaluation across speech, paralinguistic, and cultural dimensions. Its construction combines diverse speech sources with newly introduced Persian-specific tasks.

  • Benchmark overview: The benchmark totals 8,000 samples across 16 tasks: 5,000 for speech understanding, 1,500 for paralinguistic analysis, and 1,500 for Persian cultural audio understanding.Ten tasks are newly introduced specifically for Persian evaluation.
  • Benchmark overview: PARSA-Bench uniquely targets a low-resource language with a rich cultural heritage and includes ten tasks without prior equivalents in any language.These tasks capture phenomena such as poetry meter, Dastgah classification, and pragmatic register.
  • Benchmark overview: The benchmark uses consistent metrics and prompting protocols to enable fair cross-model comparison across all evaluation dimensions.This unified design is presented as a distinguishing feature from existing frameworks.
  • Data sources: Persian speech samples come from Common Voice and ParsVoice, selected across diverse speaker demographics and acoustic conditions.The sampling design aims to make ASR evaluation reflect realistic variation rather than a narrow recording environment.
  • Data sources: Bidirectional Persian-English translation uses aligned speech-translation pairs from CoVoST2 spanning diverse topics and speaking styles.The dataset supplies speech and translation supervision for the translation task.

Persian Cultural Audio Understanding

The Persian Cultural Audio Understanding dimension covers intent and entity tasks, registers, code-switching, comprehension, paralinguistic attributes, poetry, and traditional music. Its datasets combine natural recordings, annotated corpora, TTS audio, and culturally specific sources.

  • Speech and language tasks: Intent Detection and Named Entity Recognition use MASSIVE labels for Persian, with TTS audio synthesized from diverse speaker profiles and manually checked for naturalness.The construction uses Common Voice reference voices and verifies a random subset of 50 synthesized samples.
  • Speech and language tasks: Formal/Informal Register Detection draws equally from annotated formal and informal Persian speech in Mana-TTS.The source covers varied domains and social contexts.
  • Speech and language tasks: Code-Switching Detection combines spontaneous Common Voice examples with Persian YouTube recordings that naturally include English technical terms and expressions.The two sources capture scripted and naturalistic switching behavior.
  • Speech and language tasks: Reading Comprehension and QA convert ParsiNLU questions, Wikipedia passages, and short stories into audio with varied speaker characteristics.GPT-4o-mini generated plausible distractors for multiple-choice questions to reduce surface-level answer cues.
  • Paralinguistic analysis: Age and Gender Recognition uses balanced Common Voice metadata, while Emotion Recognition uses balanced acted speech from SHEMO across six emotion categories.Both tasks are constructed to support balanced paralinguistic evaluation.
  • Persian cultural audio: Persian Poetry Analysis uses Ganjoor recitations to evaluate ten frequent vazn categories and distinguish poetic styles using excerpts with sufficient structural context.The meter task has approximately 50 balanced samples per class and a random-baseline F1 of 0.10.
  • Persian cultural audio: Persian Music Understanding evaluates Dastgah classification, traditional instrument recognition, and tempo using a dataset annotated with modal, instrument, and tempo information.The Dastgah setup includes major modes such as Shur, Homayoun, Segah, and Chahargah.

4. Experiments

The experiments evaluate eight Persian-capable LALMs under zero-shot audio conditions, alongside text-only comparisons, across all 16 PARSA-Bench tasks. Performance is strongest on lexically grounded speech tasks and weakest on culturally grounded audio tasks, with no model leading across every dimension.

  • 4.1. Evaluated Models and Inference Protocol: Eight state-of-the-art LALMs were evaluated primarily in zero-shot audio, with few-shot, chain-of-thought, and text-only conditions added for comparison.The text-only condition uses transcripts to isolate audio-processing failures from language-comprehension failures.
  • 4.2. Zero-Shot Performance: The evaluation covers all 16 PARSA-Bench tasks, whose zero-shot audio performance is summarized using F1.
  • 4.2. Zero-Shot Performance: Models perform strongest on speech-understanding tasks with high lexical content, moderately on pragmatic classification, and weakest on culturally grounded audio tasks.
  • 4.2. Zero-Shot Performance: No single model dominates across speech understanding, paralinguistic analysis, and cultural audio understanding.
  • 4.2. Zero-Shot Performance: Qwen3-Omni-30B is the strongest open-source model overall, while proprietary models, particularly Gemini-2.5-Flash, lead on translation and intent detection.
  • 4.2. Zero-Shot Performance: All models perform near or below the random baseline on Persian poetry meter detection, regardless of scale or closed-weight training.

4.3. The Audio–Text Gap

The audio–text comparison separates language competence from the ability to decode speech. Most tasks show an audio disadvantage, but poetry style classification is the exception, indicating that recitation conveys information absent from text.

  • 4.3. The Audio–Text Gap: PARSA-Bench compares zero-shot audio with text-only performance for Qwen3-Omni-30B to isolate the cost of audio processing.
  • 4.3. The Audio–Text Gap: Reading comprehension and code-switching have small audio–text gaps, whereas named entity recognition and Persian-to-English translation have the largest gaps.The larger gaps reflect difficulty precisely transcribing Persian named entities and rendering speech fluently across languages.
  • 4.3. The Audio–Text Gap: The largest gaps are attributed to transcription failures rather than reasoning failures because the underlying language competence remains intact.
  • 4.3. The Audio–Text Gap: Poetry style classification is the only task where the best model performs better with audio than text-only input.Prosodic and vocal features in recitation provide style-discriminative information absent from bare text.

4.4. Paralinguistic Analysis

Paralinguistic performance forms a clear hierarchy: gender recognition is largely solved, emotion recognition remains partial, and age recognition is effectively unsolved. Cultural poetry analysis separates into near-random meter detection and substantially stronger style classification.

  • Paralinguistic tasks: Gender recognition is largely solved, with Qwen models achieving near-perfect scores regardless of scale.Gemma-E2B is the notable exception, collapsing to chance and suggesting a capability threshold at very small model sizes.
  • Paralinguistic tasks: Emotion recognition is partially solved, but all models remain well below ceiling on the six-class task.The best models perform meaningfully above the random baseline, while fine-grained affective perception remains difficult.
  • Paralinguistic tasks: Age recognition is effectively unsolved, with every model scoring near the random baseline regardless of scale or training regime.The task documents a performance ceiling for age estimation from voice alone.
  • Persian cultural audio understanding: Vazn detection is the benchmark’s most challenging task, with all models performing near chance on ten-class poetry-meter classification.Short vowels absent from standard Persian text make the required rhythmic and prosodic information unavailable to text-only pretraining.
  • Persian cultural audio understanding: Poetry style classification is substantially more tractable, and the best model’s audio performance exceeds its text-only performance.Qwen models benefit from text-side knowledge of Persian literary styles, while vocal recitation contributes discriminative information absent from transcripts.

4.6. Speech Understanding

Speech-understanding results show strong model and direction asymmetries, with audio decoding quality often limiting downstream comprehension and transcription precision. Larger models generally help, but architecture and training-data advantages can outweigh parameter count for particular tasks.

  • ASR: Qwen3-Omni-30B achieves the strongest Persian ASR performance, while smaller Gemma models have WER scores more than an order of magnitude higher.The results suggest a sharp capability threshold around the 7B parameter scale for reliable Persian ASR.
  • Speech Translation: English-to-Persian translation consistently outperforms Persian-to-English across models.The passage attributes this asymmetry to greater English-target training-data abundance.
  • Speech Translation: Gemma-E4B outperforms Qwen2.5-Omni-7B on Persian-to-English translation despite having roughly half the parameters.The passage suggests architectural or data advantages for this direction that do not generalize across tasks.
  • Audio NER: The best zero-shot audio NER model achieves only a modest F1-exact score compared with a much stronger text-only baseline.The gap between F1-exact and F1-type indicates that models often identify entity type and span but fail to transcribe Persian entity values precisely.
  • Passage Comprehension: Smaller models perform competitively on the novel TinyStories task once transcription is feasible.This suggests passage-grounded comprehension is primarily gated by audio decoding quality rather than model scale.

4.7. Prompting Strategy Analysis

Prompting results indicate that the best strategy depends on task complexity: chain-of-thought helps difficult extraction, while few-shot examples often hurt and reasoning can add noise to simple tasks. Across model-scale comparisons, performance rises within the Qwen family but does not track parameter count consistently across architectures.

  • 4.7. Prompting Strategy Analysis: Table 7 summarizes prompting-strategy effects on Qwen3-Omni-30B across representative tasks.The evaluated conditions include zero-shot, few-shot, chain-of-thought, and text-only prompting.
  • 4.7. Prompting Strategy Analysis: Chain-of-thought prompting benefits complex extraction tasks such as NER and intent detection.The passage contrasts these gains with harms on simpler binary or structural tasks.
  • 4.7. Prompting Strategy Analysis: Gemma-E4B outperforms Qwen2.5-7B on Cultural Audio despite having fewer parameters, indicating that training-data coverage matters more than model size for culturally grounded tasks.
  • 4.7. Prompting Strategy Analysis: Chain-of-thought prompting harms code-switching and formal/informal-register tasks, where reasoning traces introduce noise rather than clarity.
  • 4.7. Prompting Strategy Analysis: Few-shot prompting generally degrades performance.The passage attributes this likely to the computational cost of processing audio demonstrations and possible format confusion outweighing example benefits.
  • 4.7. Prompting Strategy Analysis: The text-only condition is almost i
  • 4.8. Impact of Model Scale: Scale alone is not a reliable predictor of performance across open-source models.Within the Qwen family, performance scales consistently across dimensions from 3B to 30B, but this pattern does not hold across architectures.
  • 4.8. Impact of Model Scale: Qwen3-Omni-30B is the strongest single model overall, but its advantage is far from uniform across tasks and dimensions.Gemma-E4B matches or exceeds Qwen2.5-Omni-7B on Cultural Audio and outperforms it on Persian-to-English translation.

5. Conclusion

The paper introduces PARSA-Bench as a comprehensive evaluation of large audio-language models on Persian language and culture. The benchmark spans 16 tasks and over 8,000 samples, and the evaluation identifies audio processing as the dominant bottleneck across dimensions.

  • PARSA-Bench is presented as the first comprehensive benchmark for large audio-language models on Persian language and culture.
  • The benchmark spans 16 tasks and over 8,000 samples across speech understanding, paralinguistic analysis, and Persian cultural audio understanding.
  • The evaluation of eight state-of-the-art large audio-language models identifies audio processing as the dominant bottleneck across all dimensions.

6. Limitations

PARSA-Bench has several scope and evaluation limitations, including synthetic audio, missing human baselines, restricted prompting coverage, and the absence of fine-tuned or retrieval-augmented systems.

  • Six tasks use TTS-synthesized audio, which may underrepresent natural speech prosodic variability and disfluency.The authors suggest complementing these tasks with fully naturalistic recordings.
  • The benchmark lacks human performance baselines, especially important for interpreting near-chance model results on vazn detection.Human judgments would provide a meaningful upper bound for culturally grounded tasks.
  • Extended prompting experiments were limited to open-source models because proprietary-model API costs prevented equivalent analysis for Gemini and GPT-4o.The authors identify fuller prompting analysis across those models as future work.
  • All models were evaluated only in zero-shot and prompting-only regimes, leaving fine-tuned and retrieval-augmented systems untested.Such systems may produce substantially different results, particularly on culturally specific tasks.
  • PARSA-Bench is intentionally scoped to Persian, so extending the framework to other low-resource languages remains future work.
Loading 2603.14456v1…