Source-linked AI summary
CARES: A Controlled Synthetic Benchmark of Speaker Reactions to Sound
Marcel Gibier, Thomas Thebaud, Olivier Boëffard, Jean-François Bonastre
TL;DR
Audio scene description lacks a reliable ground truth for which sounds matter in a conversation. CARES fixes sound salience through audible speaker reactions and generates 10,000 two-speaker scenes. Models recognize sounds, but their reaction classifications are close to text-only performance.
Problem
Descriptions must select which audio elements to retain, but annotators agree only weakly on salient content and datasets do not label which sounds mattered to participants.
Source: §1 · p. 1Method
CARES defines an audio event as salient when a speaker audibly reacts, fixes that ground truth before generating scenes, and provides 10,000 two-speaker dialogues.
Source: §1 · p. 1Results
Models classified scenes and events, but the best reaction balanced accuracy was 39.0% versus 36.5% for text, and muting sounds did not significantly change audio-language model accuracy.
Source: §7 · p. 4Takeaways & Limitations
The benchmark identifies linking sounds to what speakers do with them as an open problem, beyond recognizing sounds.
Source: §8 · p. 4The scenes are synthetic English dialogues checked on transcripts rather than audio, and model collapse may reflect audio or task format.
Source: §8 · p. 4
Abstract
from arXiv · showhide
Automatic audio scene description turns a recording into a text account of a situation. One difficulty is deciding which elements of the audio should be kept, since a description cannot include them all. Annotators disagree about this, making a ground truth hard to obtain. In this work, we first define the ground truth, then generate the data. We focus on audio events and define sound salience with a simple rule: a sound is salient when a speaker audibly reacts to it. For scale and variety, a controlled set of scenarios fixes the ground truth, and a language model writes the dialogues. The resulting corpus, CARES, contains 10,000 two-speaker scenes. We then benchmark six audio-language models on three tasks: identifying the scene, tagging the sounds present, and classifying reactions. We show that these models hear the sounds but miss how the speakers react to them.
1. INTRODUCTION
Audio scene description must select which elements of a mixed recording to retain, but salience judgments lack reliable agreement and a clear ground truth. CARES defines audio-event salience through audible speaker reactions, then uses generated scenes to test whether audio-language models capture those reactions.
1. INTRODUCTION
- AASD combines speech, events, ambience, and music, yet descriptions must select what to retain and annotators agree only weakly on salient content.The relevant salience is worth reporting, not merely perceptual prominence: a quiet sound that interrupts conversation may matter more than the loudest sound.Source: §1 · p. 1
- Prior systems either enrich transcripts without weighting content or summarize speech by importance, while treating environmental sounds as side information.Recent audio-language models take the mixed stream as input, but existing datasets do not record which sounds mattered to participants.Source: §1 · p. 1
- CARES defines an audio event as salient when a speaker audibly reacts to it, fixing the ground truth before generating scenes.The definition is restricted to audio events, not what is said.Source: §1 · p. 1
- CARES contains 10,000 two-speaker scenes, and its salience is recoverable from transcripts by human annotators and an independent model.Annotators reached κ = 0.86 on 200 scenes.Source: §1 · p. 1
- Audio-language models recognize sounds but classify speaker reactions no better than a text-only system.This benchmark addresses both whether the scenes can be generated with known salience and whether models use that salience.Source: §1 · p. 1
2. RELATED WORK
Prior work describes speech or sound components separately, or adds sound as context for a system’s response. CARES instead labels whether speakers themselves notice and react to sounds.
2. RELATED WORK
- Audio captioning describes short clips, sound-event detection and scene classification label sounds, and meeting and dialogue summarization focus on speech [15].Synthetic conversations have also been mixed with background sounds to make audio realistic [16].Source: §2 · p. 2
- Synthetic soundscape generation can fix sound classes and onset times by mixing isolated events, as in Scaper.This reverses annotation order for sound labels, but does not establish speaker reactions.Source: §2 · p. 2
- CASU [22] asks how sounds provide evidence for interpreting speech, but does not label whether speakers notice or react to them.Other audio-language and dialogue benchmarks test existing recordings or add sounds for response context [19].Source: §2 · p. 2
- Conversation analysis studies noticing events and adapting talk, while rich transcription marks disfluencies but not their link to sounds; CARES labels that link at scale.Source: §2 · p. 2
3. METHOD
CARES fixes scenarios and reaction labels before generating dialogue and audio, using language models only for components that require natural language. Its design links each event to a defined form of speaker conduct.
3. METHOD
- The method generates varied two-speaker scenes with salience known before the audio exists, using a language model only for natural-language components.Source: §3 · p. 2
3.1. Templates and scenarios
- Each template yields scenarios that specify speakers, their relation and genders, location, topic, events, reaction types, and registers.Source: §3.1 · p. 2
3.2. Speaker reactions to acoustic events
- Pivot events reorganize conversation, verbal events are mentioned briefly, behavioral events disrupt speech without being named, and ambient events have no effect.Registers specify ways of reacting; the allocator enforces uniform distributions across scenario dimensions rather than letting the language model concentrate on familiar modes [25].Source: §3.2 · p. 2
3.3. Dialogue generation
- Dialogue prompts require distinct reactions to reacted-to sounds, no reactions to ambient sounds, and behavioral breaks in the first sentence after a sound.A script checks automatically verifiable rules; optional vocalization tags are rendered in audio.Source: §3.3 · p. 2
3.4. Audio rendering
- Procedural audio mixes speech, event samples, and ambience, preserving voice timing while giving events clear audible windows and preventing levels from encoding reaction type.Behavioral sounds end just before speech resumes and can duck the reacting voice by up to 4 dB.Source: §3.4 · p. 2
4. IMPLEMENTATION DETAILS
The implementation combines deterministic scenario allocation and reaction recovery with generated dialogue and procedural audio. The corpus spans many scenarios, sounds, and locations, with template-separated splits and reserved test recordings.
4. IMPLEMENTATION DETAILS
- The 1,000 templates are split 80/10/10 for train, validation, and test, with all scenarios from each template kept in one split; the test set has 1,000 scenes and is evaluated zero-shot.Source: §4 · pp. 2–3
- The corpus comprises 1,000 templates and 10,000 scenarios spanning 24 pairs of adult roles and 13 locations, with about 770 scenarios per location and 219 different sounds.Source: §4 · p. 2
- The pipeline fixes scene and event design deterministically, uses language models for topics and dialogue, and regenerates dialogue until an independent evaluator recovers intended reactions.Source: Figure 2 · p. 3
- Different model families generate topics and dialogue and evaluate reactions, while test audio uses event takes and ambiences reserved apart from train and validation.Rendering uses fixed loudness targets and exports 44.1 kHz mono 16-bit WAV audio.Source: §4 · p. 3
- The released corpus contains 10,000 scenes averaging 120.4±18.8 s, with scenes, relations, events, and compositions uniformly distributed by construction.Source: §4 · p. 3
5. VALIDATION
Validation checks that generated dialogues preserve their assigned reactions, that people can recover the labels, and that added sounds leave speech intelligible.
Validation goal
- The validation aims to ensure each dialogue preserves its intended reactions and that the final audio mix keeps the dialogue intelligible.The allocator fixes each sound’s reaction type, but the generated dialogue may not express it as intended.Source: §5 · p. 3
Reaction-recovery filter
- The independent evaluator checks the transcript’s topic and each sound’s reaction label; dialogues failing either criterion are regenerated.The criterion requires exact agreement on reaction labels and a topic-embedding score above τ.Source: §5 · p. 3
Topic threshold
- τ is set to 0.82 at the max-Youden point, separating loose reformulations of gold topics from other scenarios’ topics within the same template.The reported ROC AUC is 0.861.Source: §5 · p. 3
Human validation
- κ = 0.86 agreement with released labels shows people can recover the four reaction categories from transcripts, not only the assigning model.Three annotators labelled 200 test scenes; their mutual agreement was κ = 0.81.Source: §5 · p. 3
Speech intelligibility
- 3.3% word error rate (95% CI [2.8, 3.9]) shows the final mix leaves dialogue intelligible, close to Whisper’s 2.7% on clean read speech.Whisper-large-v3 transcribed the mix and the synthesis scripts served as reference.Source: §5 · p. 3
6. BENCHMARK
CARES enables testing whether audio–language models connect sounds to speakers’ reactions and whether audio helps beyond words. The benchmark compares six open-weight models with a transcript-based control.
Benchmark purpose
- CARES makes it possible to test sound–reaction links because each sound’s reaction is known before the audio is generated.The paper notes that no prior dataset could test whether models relate a sound to how speakers react to it.Source: §6 · p. 4
Evaluated tasks
- The benchmark asks whether models classify reactions and whether audio helps beyond words, using text questions on 1,000 held-out test scenes.Test templates and recordings do not appear in training or validation.Source: §6 · p. 4
Evaluated models
- Six recent open-weight audio–language models are compared with a Whisper transcription and text-model control that never hears the sound.The open-weight restriction allows scores to be reproduced from released audio.Source: §6 · p. 4
7. RESULTS
Models often classify scenes and sounds, but their reaction classification is close to a text-only control and does not significantly change when the annotated sound is muted. They also rarely retain audio events in summaries.
Sound recognition
- 48%–90% scene-classification accuracy and 35%–58% audio-tagging accuracy show sound recognition, but text-only results are also substantial.Qwen3-Omni tops both audio tasks; text-only results are 65%±3% for scene classification and 31%±2% for audio tagging.Source: §7 · p. 4
Reaction classification
- 39.0% balanced accuracy for the best audio model is close to the text control’s 36.5% in reaction classification.Every audio–language model totally ignores at least one reaction class, while the text control ignores none.Source: §7 · p. 4
- No audio–language model’s balanced accuracy moved significantly when the annotated sound was muted, indicating reliance on speech rather than the event.The comparison used the same scenes with the annotated sound muted.Source: §7 · p. 4
Sound in summaries
- For events that become the topic of conversation, the best summary model retains 22%, while the text-only control retains 16.5%.Models retain at most 3.3% of unnamed sounds, compared with about 1% for the control, and retention does not increase when a sound breaks the talk.Source: §7 · p. 4
8. CONCLUSION
CARES defines sound salience through speaker reactions and shows that audio–language models recognize sounds but link them to reactions no better than a text-only system. The conclusions are bounded by synthetic English dialogues checked on transcripts rather than audio.
8. CONCLUSION
- Audio–language models recognize sounds but classify speaker reactions no better than a text-only system, and their summaries rarely mention sounds nobody named.The open problem is linking recognized sounds to what the talk does with them.Source: §8 · p. 4
- CARES fixes sound salience through audible speaker reactions and contains 10,000 two-speaker scenes.The paper identifies the unresolved task as linking a sound to how speakers react to it.Source: §8 · p. 4
- The evidence is limited to synthetic English dialogues checked on transcripts rather than audio, and for all but Qwen3-Omni the source of model collapse remains unclear.Language models write and check the dialogues and judge summaries, which may favor reactions they read easily.Source: §8 · p. 4