Source-linked AI summary
AQAScore: Evaluating Semantic Alignment in Text-to-Audio Generation via Audio Question Answering
Chun-Yi Kuan, Kai-Wei Chang, Hung-yi Lee
TL;DR
Text-to-audio evaluation needs finer semantic and compositional assessment than embedding similarity alone provides. AQAScore uses audio-aware large language models to verify targeted semantic queries probabilistically, and it consistently aligns more closely with human judgments while scaling with backbone capability. Its current scope centers on general text–audio relevance and remains limited on some specialized text-selection benchmarks.
Problem
Embedding-similarity metrics measure coarse text–audio relevance but are limited for fine-grained semantic alignment and compositional reasoning.
Method
AQAScore reformulates evaluation as probabilistic semantic verification, using an audio-aware large language model’s response probabilities to targeted queries.
Results
AQAScore consistently shows stronger alignment with human judgments across relevance, pairwise, and compositional evaluations, with Qwen2.5-Omni-7B reaching LCC 0.544 versus 0.443 for its 3B variant.
Takeaways & Limitations
AQAScore provides a backbone-agnostic evaluation framework that captures fine-grained inconsistencies and benefits from stronger underlying audio-language models.
Takeaways & Limitations
The evaluation primarily targets general text–audio relevance, while stylistic consistency and music relevance remain underexplored.
Abstract
from arXiv · showhide
Although text-to-audio generation has made remarkable progress in realism and diversity, the development of evaluation metrics has not kept pace. Widely-adopted approaches, typically based on embedding similarity like CLAPScore, effectively measure general relevance but remain limited in fine-grained semantic alignment and compositional reasoning. To address this, we introduce AQAScore, a backbone-agnostic evaluation framework that leverages the reasoning capabilities of audio-aware large language models (ALLMs). AQAScore reformulates assessment as a probabilistic semantic verification task; rather than relying on open-ended text generation, it estimates alignment by computing the exact log-probability of a "Yes" answer to targeted semantic queries. We evaluate AQAScore across multiple benchmarks, including human-rated relevance, pairwise comparison, and compositional reasoning tasks. Experimental results show that AQAScore consistently achieves higher correlation with human judgments than similarity-based metrics and generative prompting baselines, showing its effectiveness in capturing subtle semantic inconsistencies and scaling with the capability of underlying ALLMs.
1 Introduction
AQAScore addresses limitations in embedding-similarity metrics by reframing text-to-audio alignment as targeted audio question answering. Across relevance, pairwise, and compositional benchmarks, it is evaluated for correlation with human judgments and sensitivity to semantic errors.
- Motivation: Embedding-similarity metrics capture coarse relevance but struggle with multiple events, temporal order, fine-grained attributes, and compositional semantic errors.Prior analyses specifically identify difficulty distinguishing semantically inverted or attribute-swapped audio–text pairs.
- Approach: AQAScore uses audio-aware large language models to provide more fine-grained and controllable audio–text alignment assessments.The framework treats evaluation as an audio-to-text understanding task rather than relying only on feature proximity.
- Approach: Given audio and text, AQAScore asks whether the audio contains the described sound events and uses the probability of a “Yes” response as a continuous semantic-consistency measure.This probabilistic response measure is intended to detect missing or incorrect elements that global similarity scores may overlook.
- Evaluation: AQAScore is evaluated on human-rated relevance, pairwise comparison, and compositional reasoning benchmarks.These settings assess alignment with human judgments and reasoning over multiple sound events.
- Findings: The framework reports higher correlation with human judgments and greater sensitivity to semantic errors than the evaluated similarity-based approaches.The reported evaluation focuses on fine-grained semantic alignment across complementary benchmark settings.
2 Related works
Related work highlights a gap between progress in text-to-audio generation and reliable semantic evaluation. AQAScore extends question-answering-based multimodal evaluation to audio through probabilistic verification of targeted semantic queries.
- Evaluation gap: Text-to-audio systems increasingly generate realistic and diverse audio, but verifying faithful semantic correspondence remains challenging.Existing evaluation methods largely rely on embedding similarity.
- Prior question-answering evaluation: VQAScore shows that question answering can evaluate image–text alignment beyond simple score prediction.This provides a visual-domain precedent for reasoning-capable multimodal evaluators.
- Audio-aware evaluators: Audio-aware large language models have been explored as judges for audio and speech evaluation tasks.The cited applications include speech quality assessment and paralinguistic analysis.
- AQAScore: AQAScore reformulates audio–text evaluation as probabilistic verification by extracting the exact log-probability of a “Yes” response to targeted semantic queries.The paper reports this as a more sensitive and continuous metric than direct generative prompting.
3 Methods
AQAScore converts an audio–text pair into a targeted yes/no question and derives alignment from the model’s relative response probabilities. It also supports pairwise preference decisions by comparing scores across candidate audios or texts.
- Semantic verification: For an audio–text pair, the method constructs a question asking whether the audio contains the sound events described by the text.The AQA model assigns conditional probabilities to “Yes” and “No” responses.
- Score computation: AQAScore is the softmax-normalized probability of the “Yes” response relative to the “No” response.The resulting value is interpreted as the model’s confidence that the audio satisfies the textual description.
- Pairwise comparison: For two audio candidates sharing one text prompt, the method compares their individual AQAScores to determine which audio better aligns with the text.This corresponds to evaluating text-to-audio systems under a shared prompt.
- Pairwise comparison: For one audio paired with two textual descriptions, the method compares AQAScore(a, t1) and AQAScore(a, t2) to select the better description.In both pairwise settings, the higher score determines the predicted preference for comparison with human annotations.
4 Experimental Setups
The experiments evaluate AQAScore with multiple ALLM backbones across human-aligned relevance, pairwise preference, and compositional reasoning settings, comparing it with similarity and prompting baselines.
- Backbone Models: AQAScore is evaluated using Qwen2.5-Omni 3B and 7B models and the 7B Audio Flamingo 3 model.Audio Flamingo 3 is evaluated in base, chat, and think versions.
- Benchmark Datasets: Human-rated relevance is measured with RELATE and PAM against scores or preferences reflecting human judgments.RELATE uses 0–10 relevance scores, while PAM provides 1–5 human preference ratings.
- Benchmark Datasets: Pairwise evaluation tests whether metrics select human-preferred audio using RELATE-Pair and Baton-Pair.RELATE-Pair retains pairs whose human scores differ by more than two points; Baton-Pair pairs preferred and rejected clips for the same prompt.
- Benchmark Datasets: Compositional reasoning is assessed with CompA tasks targeting acoustic-event order and attribute binding.These tasks distinguish reversed event sequences and swapped attributes across events.
- Baseline Methods: CLAPScore provides a similarity-based baseline using audio and text embeddings, while ALLM prompting baselines rate relevance or choose between alternatives.The experiments report Pearson, Spearman, and Kendall correlations for ratings, and preference accuracy for pairwise comparisons.
5 Experimental Results
Across human-rated relevance, pairwise comparison, and compositional reasoning benchmarks, AQAScore generally outperforms similarity and prompting baselines. Its performance is robust to question-template variation and stronger ALLM backbones improve results.
- Human-rated Text Relevance: AQAScore generally achieves higher correlation with human ratings on RELATE and PAM than CLAPScore and other baselines.The comparison uses Pearson, Spearman, and Kendall coefficients.
- Human-rated Text Relevance: AQAScore outperforms direct-prompting and cascading baselines, which provide less consistent and human-aligned relevance estimates.The proprietary-model comparison uses direct score prediction because their log probabilities are unavailable.
- Pairwise Comparison: 77.6% pairwise accuracy is achieved by Qwen2.5-Omni-7B AQAScore on RELATE-Pair, exceeding all CLAPScore variants.AQAScore also reports higher accuracy and AUC than CLAPScore on Baton-Pair and Baton.
- Compositional Reasoning Analysis: 67.0% in Order-Text versus 52.0% in Order-Audio shows that selecting the correct audio is harder than selecting the correct caption.The Text setting selects a caption given audio, whereas the Audio setting selects audio given a caption.
- Compositional Reasoning Analysis: Attribute tasks yield lower accuracy than Order tasks, while Qwen2.5-Omni-7B AQAScore still outperforms specialized CompA-CLAP in most settings.Binding attributes such as associating a laugh with a baby or woman is described as more difficult than tracking temporal order.
- Prompt Robustness: AQAScore is relatively insensitive to question-template and minor formatting changes, indicating robustness to prompt wording.Performance means and standard deviations are reported across templates; quotation-mark changes produce negligible differences.
6 Discussion
AQAScore is presented as backbone-agnostic while benefiting from stronger ALLMs, with Qwen2.5-Omni-7B outperforming its 3B counterpart. The framework remains sensitive to how backbone tuning affects probability calibration.
- Backbone Scaling: Qwen2.5-Omni-7B outperforms Qwen2.5-Omni-3B in LCC, 0.544 versus 0.443, and pairwise accuracy, 77.6% versus 75.0%.These results support scaling with underlying ALLM capability.
- Prompt Robustness: Low variance across benchmarks and metrics indicates stable Qwen2.5-Omni-7B judgments across question templates.Figure 2 uses error bars for standard deviation over prompt variations.
- Backbone Scaling: AQAScore generalizes across ALLM architectures, while stronger backbones consistently provide better evaluation performance.The discussion characterizes the framework as backbone-agnostic and benefiting from model-capability improvements.
- Backbone Tuning: AF3-Chat reaches 24% on CompA-Order Text, whereas AF3-Think reaches 42.25%, indicating a tuning-related performance trade-off.The discussion associates this degradation with less-calibrated probability distributions for fine-grained log-probability extraction.
7 Conclusion
AQAScore evaluates text–audio alignment through targeted semantic verification rather than surface-level similarity. Across diverse settings, it aligns more strongly with human judgments and detects fine-grained inconsistencies, while scaling with stronger audio–language models.
- AQAScore probes whether audio satisfies semantic conditions implied by text, rather than relying on global embedding proximity or open-ended generation.
- AQAScore shows stronger alignment with human judgments across human-rated relevance, pairwise preference, and compositional reasoning tasks.
- AQAScore is sensitive to fine-grained inconsistencies involving temporal order and attribute binding.
- Stronger ALLMs yield more reliable and perceptually grounded AQAScore evaluation signals.
Limitations
The evaluation remains limited in perceptual coverage, human-labeled data availability, and consistent performance across some text-selection settings. The authors identify broader datasets and specialized evaluation-oriented models as directions for improvement.
- AQAScore primarily evaluates general text–audio relevance, leaving stylistic consistency and generated-music relevance underexplored.
- The analysis is constrained by limited datasets with human-labeled relevance ratings, including RELATE and PAM.
- AQAScore does not consistently outperform existing methods on the FENSE and BRACE text-selection benchmarks.
- General-purpose ALLMs may still encounter consistency issues in certain pairwise comparison tasks.
A Appendix
The appendix introduces the system prompts and instructions used throughout the experiments, covering AQAScore, prompting, and cascade setups.
- The appendix documents system prompts and instructions used in all experiments.
- These materials include the AQAScore approach, prompting setup, and cascade setup.
A.1 Prompt Templates Used in AQAScore
This appendix section presents the prompt templates used for AQAScore and the evaluated benchmark settings.
- Separate prompt instructions are provided for RELATE, PAM, FENSE, BRACE, pairwise settings, and CompA.
A.2 Prompt Templates Used in Prompting Setup
The prompting setup directly supplies the audio, system prompt, and instruction to the model, which then answers according to that instruction. Prompt templates are specified for relevance, pairwise comparison, and related benchmark tasks.
- The model receives the audio input, system prompt, and instruction directly, then answers according to the instruction.
- Prompt templates are provided for text–audio relevance experiments on RELATE and PAM.
- Templates also cover pairwise experiments on RELATE-Pair, Baton-Pair, FENSE, BRACE, and Baton.
A.3 Prompt Templates Used in Cascade Setup
The appendix documents cascade prompting, benchmark-specific templates, AQAScore question variants, model-comparison resources, and inference settings. It also describes the pairwise caption benchmarks used to test preference matching and hallucination sensitivity.
- A.3 Prompt Templates Used in Cascade Setup: The cascade setup first generates an audio caption, then judges text–audio relevance using only that generated caption.The second stage asks whether the caption matches the target text and can assign Likert-scale relevance.
- A.3 Prompt Templates Used in Cascade Setup: Cascade prompt templates are specified for RELATE, PAM, RELATE-Pair, Baton-Pair, Baton, FENSE, and BRACE, including optional concise reasoning.
- A.4 CLAPScore Checkpoints: CLAPScore results use several publicly available pretrained checkpoints, whose identifiers and training data are listed in Table 28.
- A.5 Effect on Question Templates for AQAScore: AQAScore is evaluated with default and alternative question templates, including direct event-presence and pairwise caption-comparison questions.The templates replace description placeholders with the corresponding target text prompts.
- A.6 Inference Setups: Except for specified API calls, inference uses a single NVIDIA RTX 3090 GPU with greedy decoding and a 512-token maximum.
- A.7 Pairwise Caption Comparison Benchmarks: FENSE and BRACE: FENSE and BRACE present an audio clip with two candidate captions and require selecting the caption that best matches the audio.BRACE groups caption pairs by human- and machine-generated sources and reports pairwise accuracy for each subset.
- A.7 Pairwise Caption Comparison Benchmarks: FENSE and BRACE: BRACE-Hallucination tests subtle hallucinations by replacing nouns with plausible but semantically different alternatives, with human verification.
- A.7 Pairwise Caption Comparison Benchmarks: FENSE and BRACE: AQAScore exceeds CLAPScore overall on FENSE, while direct prompting obtains the highest accuracy; BRACE hallucination accuracy exceeds 98% in most cases versus about 90% for CLAPScore.On BRACE, AQAScore is comparable to CLAPScore overall but performs lower on the HM subset because it favors detailed event descriptions over concise human-preferred summaries.