Source-linked AI summary
AT-Differ: Dataset for Audio-Text Difference Captioning
Akihiro Fujii, Yuki Okamoto, Yuki Saito, Hiroshi Saruwatari
TL;DR
Existing audio–text alignment methods mainly score or classify alignment rather than describe specific differences. This paper proposes difference captioning, constructs AT-DIFFER, and fine-tunes an MLLM on it; jointly learning detection, difference type, and captioning is more effective than training only on captioning.
Problem
Method
The paper constructs AT-DIFFER with systematically simulated audio–text differences and captions, then fine-tunes an existing MLLM using joint difference detection, type classification, and captioning tasks.
Source: §4.1 · p. 3, §5 · p. 4Results
94.8% overall accuracy for method 2 in difference detection; methods 2 and 3 also outperformed the baseline in difference type classification and all proposed methods beat the baseline across captioning metrics.
Source: §4.3 · p. 4Takeaways & Limitations
Jointly learning difference detection, type classification, and captioning was more effective than training only on difference captioning.
Source: §5 · p. 4The initial dataset limits each audio sample to at most two sound events and each pair to at most two difference types, excluding duration, count, and spatial differences.
Source: §3.2 · p. 2
Abstract
from arXiv · showhide
Evaluating audio-text semantic alignment is important in various audio-language tasks. Existing methods mainly quantify audio-text semantic alignment using a single score and cannot explicitly describe how audio and text differ. In this paper, we propose audio-text difference captioning, a task that generates textual descriptions of differences between audio and text. We also construct a new dataset AT-DIFFER containing audio-text pairs with systematically simulated differences and corresponding difference captions. Using AT-DIFFER, we fine-tune an existing multimodal large language model to describe differences between audio and text. The experimental results show that the original model has limited ability to describe such differences, whereas fine-tuning on AT-DIFFER substantially improves this ability.
1. INTRODUCTION
Audio–text semantic alignment matters across audio-language tasks, but existing methods mainly score alignment without describing specific mismatches. The paper proposes difference captioning and constructs AT-DIFFER to train an MLLM to describe those differences.
1. INTRODUCTION
- Audio–text semantic alignment matters for text-to-audio generation, audio captioning, audio retrieval, and multimodal large language models [7] [8].Source: §1 · p. 1
- Existing alignment methods predict scores or generate score explanations, but do not establish whether the explanations accurately describe audio–text mismatches.Source: §1 · p. 1
- The paper proposes audio–text difference captioning, which generates textual descriptions of differences between audio and text.Unlike a single alignment score, these captions provide more detailed and informative explanations.Source: §1 · p. 1
- Difference captions may support evidence-based feedback for text-to-audio generation, evaluation of generated text in audio captioning, and explanations of candidate audio in retrieval.For retrieval, the explanations can help users understand candidate samples without listening.Source: §1 · p. 1
- The contributions include AT-DIFFER, an open-source dataset, and fine-tuning Qwen3-Omni on it to generate difference captions and support benchmark analysis.Source: §1 · p. 1
2. RELATED WORK
Prior work scores or classifies audio–text alignment, and related work generates cross-modal difference descriptions. The paper extends difference captioning to audio and addresses uncertainty about whether MLLM explanations accurately identify audio–text differences.
Audio–text alignment evaluation
- Alignment methods measure similarity with CLAP, predict scores correlated with human evaluations, or classify entailment, neutrality, and contradiction, rather than describe specific mismatches.Source: §2 · p. 1
Audio–text alignment types
- Audio–text alignment types including temporal order, spatial distance, count, and compositionality have been used to evaluate and train text-to-audio models and inform this dataset's construction.Source: §2 · p. 1
Cross-modal difference captioning
- Text–image difference-description approaches evaluate text-to-image generation, and this study extends that concept to audio.Source: §2 · p. 1
MLLM-based alignment evaluation
- Although Liang et al. generate explanations of cross-modal inconsistencies, those explanations were not evaluated, leaving their accuracy for audio–text difference descriptions unclear.Source: §2 · p. 2
3. CREATION OF AT-DIFFER
AT-DIFFER pairs audio and text with simulated semantic misalignments and supplies captions describing their differences. Its construction samples conditions from sound-event clips, uses predefined templates, and limits task complexity in this initial investigation.
3.1. Dataset overview
- AT-DIFFER contains audio–text pairs with simulated semantic misalignments and corresponding captions describing their differences.Texts are English, and audio is monaural 32-bit float, 22.05 kHz RIFF WAV.Source: §3.1 · p. 2
3.2. Creation process
- For each FSD50K-Solo clip, the procedure randomly selects N = 6 valid combinations of audio structures and difference types, then generates pairs and captions using predefined templates.It adds a sound event when needed and substitutes FSD50K labels into the templates.Source: §3.2 · p. 2
- The dataset uses single, simultaneous, or sequential sound-event structures and five difference types covering content and temporal mismatches, plus a no-difference label.The initial design limits each sample to two sound events and each pair to two difference types; duration, count, and spatial differences are excluded.Source: §3.2 · p. 2
3.3. Splitting and statistics
- Using the original FSD50K splits yields 149,172 training pairs, 14,466 validation pairs, and 33,642 test pairs.The creation procedure is applied independently to each split.Source: §3.3 · p. 2
4. BENCHMARKING AUDIO–TEXT DIFFERENCE CAPTIONING
The benchmark fine-tunes Qwen3-Omni to detect audio–text differences, classify their types, and caption them. Jointly learning all three tasks outperformed caption-only training, while methods 2 and 3 performed comparably.
4.1. Model architecture and fine-tuning strategies
- The model receives audio and its paired text, then generates outputs for detection, type classification, and captioning; LoRA tunes self-attention while other weights remain frozen.Method 1 supervises captioning alone; method 2 concatenates all task outputs into one loss, whereas method 3 combines separate task losses with a weighted sum.Source: §4.1 · p. 3, Figure 3 · p. 3
4.2. Experimental setup
- Evaluation compares fine-tuned models with the pretrained baseline using accuracy for detection and type classification, and multiple captioning metrics against reference captions.The captioning metrics include BLEU, METEOR, ROUGE-L, CIDEr, SPICE, SPIDEr, BERTScore, and Sentence-BERT.Source: §4.2 · p. 3
4.3. Results and discussion
- 94.8% and 94.7% overall accuracy for difference detection were achieved by methods 2 and 3, versus 80.9% for the baseline and 75.4% for method 1.Method 1’s loss does not depend on difference detection, which the authors suggest likely explains its degraded detection performance.Source: §4.3 · p. 4
- 91.7% and 91.9% overall accuracy for difference type classification were achieved by methods 2 and 3, versus 58.5% for the baseline and 20.2% for method 1.The confusion matrices cover samples annotated with at most one difference type; predictions assigning two or more types are counted as “other.”Source: §4.3 · p. 4, Figure 5 · p. 4
- All proposed methods outperformed the baseline on every difference-captioning metric, and methods 2 and 3 outperformed method 1 and both random-caption baselines.Methods 2 and 3 also scored higher than randomly generated captions across all metrics; the supplied passages do not include Table 2’s metric values.Source: §4.3 · p. 4, Table 2 · p. 4, §4.2 · p. 4
- Compared with captioning-only training, jointly learning difference detection, type classification, and captioning improved classification accuracy; method 3 performed comparably to method 2, with no clear advantage from separate loss terms.Source: §4.3 · p. 4
5. CONCLUSION
The paper introduces audio–text difference captioning and AT-DIFFER, a dataset of pairs with systematically simulated differences. Fine-tuning an MLLM on the dataset substantially improves its ability to describe differences, and joint learning of three tasks was more effective than caption-only training.
5. CONCLUSION
- The paper proposes audio–text difference captioning and constructs AT-DIFFER with audio–text pairs containing systematically simulated differences.The dataset includes corresponding difference captions.Source: §5 · p. 4
- Fine-tuning an existing MLLM on AT-DIFFER substantially improved its ability to describe differences between audio and text.Source: §5 · p. 4
- Jointly learning difference detection, type classification, and captioning was more effective than training only on difference captioning.Future work includes extending the dataset to more diverse differences and evaluating other MLLMs.Source: §5 · p. 4