Source-linked AI summary
AVSD-Scenes: A Dataset for Audio-Visual Description of Urban Scenes
Dhanunjaya Varma Devalraju, Arshdeep Singh, Mark D. Plumbley
TL;DR
Urban scene labels provide only coarse representations of environments, while rich descriptions that jointly capture auditory and visual information remain limited. AVSD-Scenes adds paired natural-language descriptions through modality-specific generation followed by LLM fusion. The descriptions improve semantic alignment and cross-modal retrieval while retaining scene-discriminative information.
Problem
Urban scene labels such as “airport” or “park” omit contextual details, activities, and audio-visual interactions, and datasets jointly describing auditory and visual information remain limited.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The descriptions provide a richer natural-language representation of urban environments than predefined scene labels and may support scene understanding, retrieval, and multimodal language generation.
Source: §1 · pp. 1–2
Abstract
from arXiv · showhide
Natural language descriptions can provide rich semantic representations of audio-visual urban scenes, yet datasets that jointly describe both auditory and visual information remain limited. In this paper, we introduce AVSD-Scenes, a paired audio-visual scene description dataset for urban environments. The dataset contains 12,291 audio-visual scene descriptions generated from the TAU Urban Audio-Visual Scenes dataset. To construct the dataset, we first generate audio- and visual-based descriptions using Qwen2-Audio-7B and Qwen2.5-VL-7B, respectively. These modality-specific descriptions are then combined using large language models, namely Qwen3-14B, Mistral-Small-3.2-24B-Instruct-2506, and Gemma-3-27B-it, to produce multimodal descriptions that capture complementary information from both modalities. We benchmark AVSD-Scenes using semantic alignment, cross-modal retrieval, scene classification, LLM-as-a-judge evaluation, and human subjective assessment. Results show that multimodal descriptions improve semantic alignment and cross-modal retrieval performance compared with modality-specific descriptions while preserving strong scene-discriminative information. The generated descriptions achieve up to 94.5% accuracy in urban scene classification, while combining audio, visual, and description embeddings further improves accuracy to 95.4%. Furthermore, the descriptions remain highly scene-discriminative even when scene labels are removed from the prompting instructions, indicating that they capture semantic information derived from the audio-visual content rather than merely reflecting label information.
1. INTRODUCTION
AVSD-Scenes addresses the limited availability of rich natural-language descriptions for urban audio-visual scenes, which predefined scene labels represent only coarsely. It introduces paired descriptions and benchmarks their semantic and scene-discriminative value.
1. INTRODUCTION
- Predefined labels such as “airport” or “park” omit contextual details, activities, and audio-visual interactions needed to describe urban scenes richly.The TAU dataset captures holistic urban environments using standardized recording protocols.Source: §1 · p. 1
- AVSD-Scenes provides 12,291 paired audio-visual scene descriptions extending the TAU Urban Audio-Visual Scenes dataset.The descriptions represent scene context in natural language beyond predefined scene categories.Source: §1 · pp. 1–2
- The automated pipeline fuses audio- and visual-specific descriptions generated by foundation models to create multimodal scene descriptions.The paper benchmarks retrieval, semantic alignment, scene classification, and description quality with LLM-based and human assessment.Source: §1 · p. 2
- Multimodal descriptions improve retrieval and semantic alignment over modality-specific descriptions, retain scene discrimination without scene-label information in prompts, and complement conventional audio-visual representations.The dataset, code, and evaluation criterion are publicly available.Source: §1 · p. 2
2. DATASET CONSTRUCTION PIPELINE
AVSD-Scenes augments synchronized urban audio-visual recordings with natural-language descriptions in a two-stage pipeline. Modality-specific events and captions are generated first, then fused into a grounded unified description.
2. DATASET CONSTRUCTION PIPELINE
- The dataset augments TAU Urban Audio-Visual Scenes 2021 development samples with natural-language descriptions of acoustic and visual events beyond predefined labels.The source dataset contains synchronized audio-visual recordings with scene-level labels, collected in 10 European cities.Source: §2 · p. 2
2.1. Stage 1: Modality-specific description generation
- Qwen2-Audio-7B-Instruct and Qwen2.5-VL-7B-Instruct generate chronological audio and visual events and concise modality-specific captions, with scene labels supplied as context.Visual instructions request observable objects and actions and discourage inferring unobservable events or intentions.Source: §2.1 · p. 2
2.2. Stage 2: Multimodal descriptions generation
- Qwen3, Mistral, and Gemma combine Stage 1 events and descriptions into a single grounded scene description, constrained to approximately 30–60 words.Instructions discourage unsupported inferences, hallucinated details, and redundancy.Source: §2.2 · p. 2
3. EVALUATION FRAMEWORK
The evaluation framework tests description-to-modality semantic alignment and retrieval, scene classification, and description quality. It specifies embedding-based comparisons, a classifier setup, and complementary LLM and human assessments.
3.1. Semantic alignment and cross-modal retrieval benchmark
- Semantic alignment and scene-level retrieval compare audio, visual, and text embeddings, using modality-specific or multimodal descriptions for text and recall at rank k for retrieval.CLIP, CLAP, and IMAGEBIND provide embeddings; cosine similarity compares clip and description representations.Source: §3.1 · pp. 2–3
3.2. Scene classification evaluation
- Scene classification evaluates description embeddings alone and concatenated audio-visual-description features on the predefined 70% training and 30% test split.Audio and visual embeddings come from L3-Net, text embeddings from bert-base-uncased, and a one-vs-rest RBF-kernel SVM performs classification.Source: §3.2 · p. 3
3.3. Description quality evaluation
- Description quality is assessed with an LLM judge across eight five-point criteria and with four participants rating Mistral descriptions across six criteria.The human assessment covers 100 randomly selected samples balanced across scene classes.Source: §3.3 · p. 3
4. DATASET BENCHMARKING AND ANALYSIS
Multimodal descriptions improve text–audio alignment and retrieval while retaining much of the visual correspondence. They also preserve strong scene-discriminative information, with further gains from combining description, audio, and visual embeddings.
Scene classification analysis
- Multimodal descriptions substantially outperform the audio, visual, and audio–visual classification baselines; adding audio and visual embeddings yields further gains.The reproduced baseline accuracies are 73.14% for audio, 69.33% for visual, and 82.19% for audio–visual.Source: Figure 3 · p. 3, §4 · pp. 3–4
Cross-modal similarity and retrieval
- Multimodal descriptions improve text-to-audio retrieval with CLAP and IMAGEBIND, while text-to-visual retrieval remains high and similar to modality-specific descriptions.The same pattern appears in similarity: text–audio similarity improves while text–visual similarity remains similar; Qwen3 has the highest average similarity among multimodal descriptions.Source: §4 · p. 3, Table 2 · p. 4
Embedding visualization
- Multimodal descriptions produce more distinct scene-category groupings in t-SNE visualizations than audio- or visual-only embeddings.This provides qualitative support for the classification findings.Source: Figure 4 · p. 4, §4 · p. 4
Description quality analysis
- Human ratings were 3.51 for audio fidelity, 4.14 for visual fidelity, 3.63 for event coverage, 3.22 for hallucination, 4.26 for fluency, and 3.36 for overall quality.Ratings broadly support the LLM assessment; every criterion remains above the rating-scale midpoint.Source: §4 · p. 4
Instance-level retrieval
- Instance-level retrieval is relatively weak, particularly text-to-audio, because descriptions capture scene-level semantics and same-class clips often contain similar content.Similar audio and visual content among clips from the same scene class makes exact instance matching more challenging.Source: Table 3 · p. 4, §4 · p. 4
Effect of scene-label conditioning
- Without scene labels in the instructions, descriptions remain strongly scene-discriminative, and combining them with audio or visual embeddings improves over corresponding baselines.Label-guided prompting achieves higher overall performance, but the no-label results retain discriminative semantics from audio and visual content.Source: Figure 6 · p. 4, §4 · p. 4
5. CONCLUSION
AVSD-Scenes provides 12,291 descriptions from the TAU Urban Audio–Visual Scenes dataset, and its multimodal descriptions improve alignment and retrieval while supporting scene classification. Descriptions remain scene-discriminative without scene labels in prompts; future work targets scaling and reduced hallucination.
Dataset and pipeline
- AVSD-Scenes contains 12,291 audio–visual scene descriptions generated from the TAU Urban Audio–Visual Scenes dataset.The pipeline combines audio and visual descriptions using LLMs to produce unified multimodal descriptions.Source: §5 · p. 4
Results
- 94.5% scene classification accuracy is achieved using text embeddings alone, increasing to 95.4% with audio and visual embeddings.The descriptions also improve semantic alignment and cross-modal retrieval while providing informative scene-level representations.Source: §5 · p. 4
Conclusion and future work
- Descriptions remain strongly scene-discriminative without scene labels in prompts, and future work will scale the dataset and reduce hallucinations.Human evaluation further supports the quality of the generated descriptions.Source: §5 · p. 4