PaperlayerView exact revision on arXiv

Source-linked AI summary

AU-Harness: An Open-Source Toolkit for Holistic Evaluation of Audio LLMs

Hoang Nguyen, Sidharth Surapaneni, Akshay Kalkunte, Jash Mehta, Aman Tiwari, Oluwanifemi Bamgbose, Khyati Mahajan, Jash Shah, Shruthan Radhakrishna, Sathwik Tejaswi Madhusudhan, Vikas Yadav, Sai Rajeswar

arXiv:2509.08031v3cs.SDcs.AIcs.LGeess.AS

TL;DR

LALM evaluation frameworks often lack throughput, reproducibility, and multi-turn coverage as models and audio benchmarks grow. AU-Harness combines configurable evaluation with coordinated, sharded inference. The paper reports improved evaluation efficiency and presents the framework as supporting reproducible, large-scale studies.

  • Problem

    Existing evaluation toolkits underuse efficient processing, lack standardized settings, and largely restrict assessment to single-turn interactions as LALMs and audio benchmarks expand.

    Source: §1 · pp. 1–2

  • Method

    AU-Harness combines a configurable evaluation framework with token-request coordination and parallel engines using dataset sharding.

    Source: §4 · p. 4, §4.1 · p. 5

  • Results

    AU-Harness’s token orchestration and data sharding improve efficiency for audio benchmark evaluations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework aims to make flexible, large-scale evaluations more reproducible and accessible for systematic LALM research and practical deployment.

    Source: §6 · pp. 9–10

    The analytical studies are representative case studies, including one model on one benchmark for multi-turn dialogue, so generalization across broader LALM families remains open.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Large Audio Language Models (LALMs) are rapidly advancing, but evaluating them remains challenging due to inefficient and non-standardized toolkits that limit fair comparison and systematic assessment. Existing evaluation frameworks exhibit three critical limitations: (1) slow and inefficient processing pipeline that bottlenecks large-scale studies, (2) inadequate multi-turn dialogue support, leaving fundamental questions about cross-turn context integration and performance dynamics over extended conversations in LALMs unanswered; and (3) the absence of unified and scalable evaluation framework capable of keeping pace with the rapid growth of both LALMs and audio benchmarks. To address these issues, we introduce AU-Harness, an efficient and comprehensive evaluation framework for LALMs. Our system achieves a speedup of up to 151% over existing evaluation toolkits through optimized batch processing and parallel execution, enabling large-scale evaluations previously considered impractical. We provide standardized prompting protocols and flexible configurations for fair model comparison across diverse scenarios. AU-Harness unlocks a range of in-depth analyses difficult to conduct without a unified foundation, including multi-turn dialogue dynamics, enabling the study of true audio reasoning capabilities in existing LALMs. AU-Harness provides both practical evaluation tools and insights into model limitations, advancing systematic LALM development.

1 Introduction

As LALMs expand beyond text into speech, sounds, and multimodal inputs, evaluation infrastructure has received comparatively little attention. AU-Harness addresses throughput, reproducibility, and multi-turn coverage with a unified, configurable framework for systematic evaluation.

1 Introduction

  • Existing toolkits underuse batching, standardize prompts inadequately, and largely omit multi-turn interactions needed to assess extended conversations.These limitations impede large-scale systematic comparisons and leave evaluation settings incomparable.Source: §1 · pp. 1–2
  • AU-Harness combines token request orchestration and data sharding to scale audio evaluations across multiple nodes and hardware architectures.The authors report that these mechanisms improve evaluation efficiency.Source: §1 · p. 2
  • AU-Harness provides comprehensive support across five evaluation dimensions, targeting systematic and reproducible LALM assessment at scale.The dimensions include inference backend integration, multi-turn support, concurrent multi-task execution, and configurable evaluation design.Source: §1 · p. 2
  • Its contributions include vLLM batching, dataset sharding, standardized prompting and metrics, and configurable multi-turn dialogue evaluation.The framework is presented as supporting in-depth analysis of dialogue behaviors over extended interactions.Source: §1 · p. 2

2 Related Work

Audio benchmarks have broadened from core task axes to instruction-following, conversational, and complex-audio tasks. Evaluation kits have received less attention, and existing offerings still leave gaps in extensibility and scalable comparison.

2 Related Work

  • Audio benchmarks expanded from core task axes to instruction-tuned and sequence-generation tasks, conversational behaviors, and complex audio reasoning.SUPERB established content, speaker, semantics, and paralinguistics axes; later benchmarks broadened coverage across speech, music, environmental audio, and complex scenes.Source: §2 · p. 2
  • Audio evaluation kits received less attention as early tasks required minimal setup, despite increasingly complex benchmarks prompting newer, broader kits.AudioBench supports up to eight tasks across 26 datasets, while VERSA focuses on text-to-audio signal quality.Source: §2 · pp. 2–3
  • Existing kits often assume one model and one benchmark per run, while AHELM’s largely fixed suite requires task-specific code to extend.The paper positions AU-Harness as a customizable kit for large-scale evaluation across LALMs and audio benchmarks.Source: §2 · p. 3

3 LALM Evaluation Challenges

The section frames evaluation challenges in throughput, conversational coverage, and customization. Existing toolkits have feature and scaling gaps, while multi-turn assessment and filtering can expose performance dimensions that static or unfiltered evaluation misses.

3 LALM Evaluation Challenges

  • Table 1 compares toolkits on inference integration, multi-turn support, concurrent multi-task processing, customization, and MELD-Emotion throughput.Throughput is reported with Samples Per Second and Real-Time Factor; the caption says AU-Harness supports most dimensions.Source: Table 1 · p. 3

3.1 Inference Efficiency

  • Single-model, single-benchmark runs and underused parallel processing constrain efficient large-scale comparisons across LALMs and audio benchmarks.RTF measures evaluation time relative to processed audio duration, while SPS counts samples processed per second; a 500-sample MELD-Emotion study found high RTF and low SPS for most existing kits.Source: §3.1 · p. 3

3.2 Customizable Evaluation Configurations

  • Multi-turn evaluation can assess dialogue continuity, contextual reasoning, and adaptation across extended conversations relevant to interactive voice assistants.Without this support, approaches risk overlooking usability and robustness aspects of realistic agentic voice systems.Source: §3.2 · pp. 3–4
  • Customizable dataset filtering supports granular analysis across audio conditions, including short-form audio relevant to dialogue-state tracking.The passage contrasts possible reliability on 10-second chunks with difficulty handling short-form audio.Source: §3.2 · p. 4

4 AU-Harness

AU-Harness combines configurable evaluation settings, coordinated inference, and task-specific parallel engines. It supports customizable metrics and dialogue histories, while allowing new tasks and datasets to be added with limited configuration changes.

4 AU-Harness

  • AU-Harness comprises a hierarchical Config module, a Central Request Controller coordinating token requests, and Concurrent Engines running task-specific evaluations in parallel.Engines can support multi-model evaluations tailored to particular tasks.Source: §4 · p. 4

4.3 Support for features and tasks

  • New task configurations can reuse a base configuration with minimal updates, and Table 2 covers representative model groups and audio-task categories.The table reports LLM-as-judge and task-specific automatic metrics; bold and underlining indicate highest and second-highest values.Source: Figure 2 · p. 5, Table 2 · p. 6

4.1 Inference Efficiency

  • A global concurrency-slot pool, adaptive synchronization, proportional dataset sharding, and native vLLM integration target scalable inference throughput.Slots are acquired before requests and released upon completion; sharding distributes disjoint subsets in proportion to endpoint concurrency capacity.Source: §4.1 · p. 5

4.2 Customizable Evaluation Configurations

  • YAML configurations specify metrics, models, aggregation, and prompt overrides without task-specific Python glue; users can also configure metrics and aggregation.AU-Harness separates inference and metric computation from hosting and supports vLLM-compatible models or standard chat-completions backends.Source: §4.2 · p. 5

4.3 Support for features and tasks

  • For multi-turn evaluation, users configure whether historical turns use original audio, reference transcripts, or model-generated transcriptions.Synchronous turn chains recursively append model outputs to dialogue context, supporting controlled evaluation across heterogeneous LALMs.Source: §4.3 · p. 6

5 Results & Discussion

AU-Harness outperforms existing evaluation kits in efficiency, including against a vLLM-backed baseline, and supports comparative analyses of instruction modality and multi-turn dialogue. Those analyses show task-dependent modality effects and declining cumulative-state dialogue performance that stabilizes when prior-state errors are controlled.

5.1 Inference Efficiency

  • Up to 151% relative improvement in SPS and 61% relative reduction in RTFs over LMMS-Eval with vLLM; total-runtime gains hold across evaluated datasets and models.Figure 3 compares the efficiency metrics across runtime scenarios, and Figure 4 shows the total-runtime comparison.Source: §5.1 · p. 7, Figure 3 · p. 7, Figure 4 · p. 7
  • In Reality parallel execution, AU-Harness outperforms LMMS-Eval by 26% in RTF relative reduction and 34% in average SPS relative gain.Against naive vLLM integration, AU-Harness also achieves an additional approximately 16% average SPS relative improvement; Table 3 compares optimal and Reality parallel runtimes.Source: §5.1 · p. 8, Table 3 · p. 8

5.2 Insights enabled by unified evaluation framework

  • On IFEval, Whisper-v3 yields 99.59 BERTScore while performance falls from 87.56 to 79.74 (∆= -7.82); the model’s own ASR yields 37.52 BERTScore and performance of 30.07 (∆= -57.49).Qwen3-Captioner yields 88.57 BERTScore but recovers performance only to 58.76 (∆= -28.80).Source: §5.2 · pp. 8–9, Table 4 · p. 8
  • On GSM8K, direct audio reasoning degrades performance by ∆= -30.93, while own ASR recovers it to 72.02 (∆= -5.31) and Whisper-v3 closes the gap (∆ = +0.46).The authors report that modality effects differ by task: instruction following is sensitive to transcription artifacts, whereas mathematical reasoning is bottlenecked at audio-grounded inference.Source: Table 4 · p. 8, §5.2 · p. 9
  • On SpokenWOZ, standard-configuration JGA approaches zero by turn 14 as accumulated slots rise, while oracle prior-state scoring stabilizes JGA.The oracle setting supplies the ground-truth prior state and scores per-turn slot deltas.Source: §5.2 · p. 9, Figure 5 · p. 9
  • Text-only inputs achieve the highest JGA in both settings; spoken current turns lower JGA, and all-audio input is substantially worse under oracle scoring.The authors report that the current-turn audio-text gap is more pronounced under oracle scoring, while the all-audio trend is ambiguous in standard evaluation.Source: §5.2 · p. 9, Figure 5 · p. 9

6 Conclusion

AU-Harness is a modular, extensible evaluation framework for LALMs that supports broad task coverage and customization. Its efficiency gains make flexible, large-scale evaluations more accessible and reproducible.

6 Conclusion

  • AU-Harness is modular and extensible, allowing researchers to customize benchmarks and integrate models or tasks without major restructuring.Its design emphasizes broad task coverage, ease of use, and adaptability.Source: §6 · p. 9
  • Dataset sharding and effective token-request orchestration realize the framework’s efficiency gains.Source: §6 · p. 9
  • The framework supports flexible, large-scale evaluations that were previously difficult to conduct reproducibly and accessibly.The authors aim to lower benchmarking barriers and foster customization.Source: §6 · p. 10

Limitations

The authors identify constraints in backend support, prompt fidelity, benchmark coverage, and the scope of their analytical studies. They also note that benchmark performance may not generalize to real-world audio contexts without further validation.

Limitations

  • Efficiency gains are evaluated with vLLM; models without mature backends may revert to conventional execution with reduced throughput.Closed-source endpoints depend on chat-completions APIs, limiting batching control and exposing runs to provider rate limits, queuing, and transient failures.Source: Limitations · p. 10
  • Standardized prompts improve reproducibility but cannot eliminate prompt sensitivity, and canonical prompts may bias open-ended task results toward specific behaviors.The authors call for multiple documented prompt families and complementary temporal measures.Source: Limitations · p. 10
  • AU-Harness is an evaluation framework, not a fixed benchmark suite; its objective is seamless integration of new benchmarks, tasks, and metrics.The authors describe wide coverage while emphasizing extensibility.Source: Limitations · p. 10
  • Coverage remains skewed toward English and common domains, with environmental audio, music understanding, and low-resource languages underrepresented.The relationship between standardized benchmark results and real-world performance in noisier, more diverse contexts requires empirical validation.Source: Limitations · p. 10
  • The two Section 5.2 analyses are representative case studies, not exhaustive evidence about broader LALM families.One examines Qwen3-Omni-30B-A3B-Thinking; the multi-turn DST study covers one model on SpokenWOZ, so generalization of the observed gaps and degradation remains open.Source: Limitations · p. 10
  • The authors identify community coordination around prompting standards, temporal diagnostics, and multilingual breadth as challenges for reproducible, comprehensive, and valid assessment.The framework is designed to enable practical, systematic progress in these areas.Source: Limitations · p. 10

Ethics Statement

The authors report attention to copyright, privacy, and speaker consent in benchmark datasets, while noting that evaluated models may still produce harmful or biased outputs. Current task coverage is skewed toward English and common domains; language models were used only to refine manuscript wording.

Ethics Statement

  • The benchmark datasets are described as respecting copyright and privacy guidelines, with particular attention to speaker consent in diarization tasks.Source: Ethics Statement · p. 10
  • AU-Harness cannot guarantee that evaluated models will avoid harmful or biased audio-related outputs, so deployment requires appropriate content filtering and bias detection.Source: Ethics Statement · p. 10
  • Current task coverage is skewed toward English and common domains, which the authors say may reinforce representational biases in audio AI.They encourage extending the framework to more diverse languages and cultural contexts.Source: Ethics Statement · p. 10
  • Language models were used only to refine manuscript language for clarity and correctness, not to generate substantial content or claims.Source: Ethics Statement · p. 11

Reproducibility Statement

The authors state that code, configurations, scripts, and documentation will be publicly released under an open-source license upon acceptance. They describe implementation and environment details intended to support replication of the evaluations.

Reproducibility Statement

  • The planned release includes AU-Harness code, configuration files, evaluation scripts, and documentation under an open-source license upon acceptance.Source: Reproducibility Statement · p. 11
  • The appendices provide hyperparameters, model endpoints, preprocessing steps, metrics, and efficiency-comparison hardware and execution settings.Documented efficiency settings include hardware specifications, vLLM versions, concurrent request limits, and retry policies.Source: Reproducibility Statement · p. 11
  • Docker containers with fixed dependency versions, setup instructions, random seeds, sampling parameters, and judge configurations are provided to support replication.The statement also lists text-to-speech synthesis parameters and prompt templates for the new reasoning benchmarks.Source: Reproducibility Statement · p. 11

A.1.1 Benchmark Details

AU-Harness brings together broad benchmark coverage, configurable evaluation conditions, and a run report for analyzing model performance. Its efficiency ablations show benefits from batching, parallel execution, and replica scaling.

A.1.1 Benchmark Details

  • The benchmark suite covers 56 datasets across six categories, with Spoken Language Understanding the largest category at 21 datasets.The categories range from Audio Understanding and Paralinguistics to Speech Recognition and Spoken Language Reasoning.Source: §A.1.1 · p. 13, Table 5 · p. 14

A.1.2 Metric Details

  • Reported metrics include WER, BLEU, IFScore, Joint Goal Accuracy, and several LLM-judge scores for correctness, safety, and multi-turn evaluation.WER is lower-is-better, while BLEU and IFScore are higher-is-better.Source: §A.1.2 · p. 14

A.1.3 Evaluated Models

A.2 Inference Efficiency Evaluation Settings

  • Efficiency comparisons use 3×H100 GPUs, disable caching, and contrast sequential execution with an idealized zero-overhead parallel runtime.The setup uses 500 samples each from MELD-Emotion, LibriSpeech-clean, and ClothoAQA, spanning 1,476–11,376 seconds of audio.Source: §A.2 · p. 16, Table 6 · p. 16

A.5 Inference Efficiency Ablations

  • 3.5× higher throughput over sequential execution; replica scaling lowers latency with near-linear improvements up to 25 replicas.Increasing batch size also reduces execution time, although benefits taper at higher scales.Source: Figure 6 · p. 17, §A.5 · p. 17

A.6 Instruction Following Experimental Settings

The AU-Harness Run Report

  • The end-of-run HTML report organizes results into seven tabs, while multi-turn plots cap evaluation at 20 turns and omit the opening turn.The cap addresses possible context-length truncation in all-audio settings; the opening turn is omitted because it is nearly always a greeting.Source: The AU-Harness Run Report · p. 19, Turn Range Selection · p. 19
Loading 2509.08031v3…