Source-linked AI summary
Listen As You Wish: Audio based Event Detection via Text-to-Audio Grounding in Smart Cities
Haoyu Tang, Yunxiao Wang, Jihua Zhu, Shuaike Zhang, Mingzhu Xu, Qinghai Zheng, Yupeng Hu
TL;DR
Sound-event detection is limited to predefined event classes, while text-to-audio grounding must match complex language queries to all corresponding segments in untrimmed audio. CGI models relations among query words, attends to fine-grained audio–query interactions, and gates cues by relevance. The authors report that it outperforms several state-of-the-art baselines on AudioGrounding.
Problem
Sound-event detection is limited to predefined event classes, while grounding complex language queries requires matching query semantics to relevant segments in untrimmed audio.
Source: §1 · p. 2Method
CGI combines a graph over query words, fine-grained cross-modal attention, and cross-gating of audio and query cues according to their relevance.
Source: §1 · pp. 2–3Results
The authors report that CGI outperforms several state-of-the-art baselines on the AudioGrounding dataset.
Source: §1 · p. 3Takeaways & Limitations
Accurate and timely traffic-event detection is a prerequisite for improving vehicle–road cooperative control.
Source: §1 · p. 1
Abstract
from arXiv · showhide
With the development of internet of things technologies, tremendous sensor audio data has been produced, which poses great challenges to audio-based event detection in smart cities. In this paper, we target a challenging audio-based event detection task, namely, text-to-audio grounding. In addition to precisely localizing all of the desired on- and off-sets in the untrimmed audio, this challenging new task requires extensive acoustic and linguistic comprehension as well as the reasoning for the crossmodal matching relations between the audio and query. The current approaches often treat the query as an entire one through a global query representation in order to address those issues. We contend that this strategy has several drawbacks. Firstly, the interactions between the query and the audio are not fully utilized. Secondly, it has not distinguished the importance of different keywords in a query. In addition, since the audio clips are of arbitrary lengths, there exist many segments which are irrelevant to the query but have not been filtered out in the approach. This further hinders the effective grounding of desired segments. Motivated by the above concerns, a novel Cross-modal Graph Interaction (CGI) model is proposed to comprehensively model the relations between the words in a query through a novel language graph. To capture the fine-grained relevances between the audio and query, a cross-modal attention module is introduced to generate snippet-specific query representations and automatically assign higher weights to keywords with more important semantics. Furthermore, we develop a cross-gating module for the audio and query to weaken irrelevant parts and emphasize the important ones.
1. Introduction
In smart-city traffic monitoring, sound-event detection is limited to predefined event classes and cannot distinguish complex event sequences, while text-to-audio grounding must match a language query to all corresponding audio segments. CGI addresses this challenge by modeling query-word relations, fine-grained audio–query interactions, and the relevance of audio and query cues.
1. Introduction
- Sound-event detection is restricted to predefined event classes and may fail to distinguish separate collisions, limiting its usefulness for complex traffic scenarios.Visual sensing can also be degraded by lighting conditions and shooting angles.Source: §1 · p. 2
- TAG must localize every segment matching a potentially complex query in untrimmed audio, including the ordering of events such as repeated collisions followed by an explosion.The query also requires the model to downweight irrelevant audio content and align snippets with query semantics.Source: §1 · p. 2
- CGI models local and non-local relations between query words, attends to fine-grained audio–query interactions, and gates audio and query cues according to their relevance.The model separately encodes audio snippets and query words before applying these components.Source: §1 · pp. 2–3
- The authors report that CGI outperforms several state-of-the-art baselines on the AudioGrounding dataset.The figure illustrates TAG grounding a series of vehicle collisions and an explosion from 16.1s to 20.3s, unlike SED’s distinction between a single collision and a series.Source: §1 · p. 3, Figure 1 · p. 3
2. Related Work
The related work covers sound-event detection, language grounding in visual data, and graph neural networks for language. CGI applies query modeling and cross-modal interaction to audio grounding, where queries may require locating multiple segments.
2.1. Sound Event Detection in Smart Cities
- Sound-event detection methods locate and classify events from a predefined set; TAG instead uses language queries, and CGI models query relations and fine-grained cross-modal interactions for improved localization.The TAG approach described encodes the whole sentence as an average-pooled global feature.Source: §2.1 · pp. 3–4
2.2. Language Grounding in Visual Data
- Unlike visual grounding methods that localize an object or moment, audio grounding may require returning multiple segments, and CGI adds cross-gating to emphasize relevant audio and query cues.The related methods cited use query attention or graph-based representations in visual grounding.Source: §2.2 · p. 4
2.3. Graph Neural Networks in Language
- Graph neural networks have been used to capture semantic relations in language; CGI uses an intra-modal query graph to propagate messages and capture high-order query relations.Source: §2.3 · p. 4
3.1. Problem Formulation
Text-to-audio grounding takes an untrimmed audio sequence and a word-by-word query, and aims to locate every ground-truth segment in the audio. Each target segment is specified by its onset and offset.
3.1. Problem Formulation
- The task input is an untrimmed audio sequence represented as snippets and a query represented word by word.Source: §3.1 · p. 4
- For each audio–query pair, the model aims to ground all ground-truth audio segments, with each segment defined by its onset and offset.Source: §3.1 · p. 4
3.2. Encoder
The encoder section describes separate audio and query feature extraction. Audio is represented with log-Mel features and encoded into temporally aligned features, while query words are represented with embeddings.
3.2. Encoder
- The audio encoder applies log-Mel spectrogram extraction, followed by a CRNN with five padded 3 × 3 convolution blocks and a bidirectional GRU; upsampling restores the original audio length.The encoded audio embeddings have length L and dimension m.Source: §3.2 · p. 4
- The query encoder maps each word to an embedding, while the model architecture includes separate audio and query encoders.Source: §3.2 · p. 4, Figure 2 · p. 5
3.3. Intra-modal Query Graph
The query graph models words as nodes connected by directed pairwise relations, and stacked graph layers enrich their representations for cross-modal interaction.
3.3. Intra-modal Query Graph
- A directed query graph treats each word as a node and includes edges between every word pair, with N stacked layers capturing intra-query relations.Each edge represents a relation from one word node to another, with a reverse edge also included.Source: §3.3 · p. 5
3.3.1. Message Passing and Aggregation
- Position encodings are appended to word-node features because neighboring words are considered more important than distant words.The position information is integrated when calculating edge weights between paired nodes.Source: §3.3.1 · p. 5
3.3.2. Update of node representations
- Each word node aggregates messages from other word nodes using edge weights before its representation is updated.The update fuses the prior node feature with received messages using a ConvGRU, and the final-layer word representations feed cross-modal interaction.Source: §3.3.1 · p. 5, §3.3.2 · p. 6
3.4. Cross-Modal Interaction
Cross-modal attention creates a query representation for each audio snippet, then cross-gating adjusts the audio and query features according to their paired relevance.
3.4.1. Attention Module
- Snippet-by-word attention weights query words by their relevance to each audio snippet, producing a snippet-specific query representation.The module contrasts with averaging all query words equally, since words such as “ambulance” and “siren” can carry more semantics for localization.Source: §3.4.1 · p. 6
3.4.2. Cross-gating Module
- Cross-gating uses each modality to gate the other, reducing the influence of unrelated audio-query pairs and enriching interactions when they are closely related.The gating weights are computed from the paired audio and query features.Source: §3.4.2 · pp. 6–7
3.5. Grounding and Learning
The grounding module estimates event on-sets and off-sets from audio-query similarity, with binary cross-entropy training and thresholded predictions.
3.5. Grounding and Learning
- The grounding module estimates audio event on-sets and off-sets using similarities between audio snippets and sentence-query features.It produces a similarity vector with the audio length.Source: §3.5 · p. 7
- Binary cross-entropy is used as the training criterion, with each audio embedding labeled for whether the query is present.The target label is 1 or 0 for query presence or absence in that embedding.Source: §3.5 · p. 7
- At evaluation, the similarity vector is binarized into predictions using threshold β.Source: §3.5 · p. 7
4. Experiment
Experiments evaluate CGI on AudioGrounding using standard event-based and threshold-independent metrics, then compare it with prior methods, ablations, and qualitative examples. CGI achieves the strongest reported overall comparison, while ablations and visualizations examine the model’s component contributions and query-graph behavior.
4. Experiment
- The experiments were conducted on AudioGrounding, with other methods’ hyperparameters and training settings kept as reported in their papers.Source: §4 · p. 7
4.1. Datasets
- AudioGrounding contains 4,590 audios, 4,994 captions, and 13,985 audio-query pairs, split into 12,373 training, 451 validation, and 1,161 test pairs.Source: §4.1 · p. 7
4.2. Implementation Details
- Queries are lowercased and tokenized; 256-dimensional word embeddings and 64-dimensional LMS audio features are used to train CGI end to end.Training uses Adam, batch size 64, at most 100 epochs, early stopping, and a 0.4 evaluation threshold.Source: §4.2 · p. 7
4.3. Evaluation Metrics
- Evaluation uses event-based precision, recall, and F1, plus threshold-independent PSDS; F1 uses a 100 ms t-collar and 20% segment tolerance.Source: §4.3 · p. 7
4.4. Comparison with State-of-the-Arts
- 2.8% absolute PSDS improvement over Match-LSTM; CGI also leads all baselines across metrics, with around 1.0% and 2.0% absolute improvements over Match-LSTM.The authors attribute the overall results to the combined effects of the query graph, cross-modal attention, and cross-gating.Source: Table 1 · p. 8, §4.4 · p. 8
4.5. Ablation Study
- Almost all variants beat every Table 1 comparator; removing the query graph especially degrades PSDS, while cross-gating and positional encoding improve nearly all metrics except CGI Precision remains competitive.The authors attribute the effects to lost implicit word relations, meaningful audio/query representations, and temporal context when those components are removed.Source: §4.5 · p. 9, Table 2 · p. 9
4.6. Qualitative Results
- The query graph gives “a” a concentrated self-weight but distributes “engine”’s edge weights more evenly, including a strong relation between “humming” and “engine.”The authors interpret these weights as bridging implicit word correspondences that enrich query features and benefit grounding.Source: §4.6 · pp. 9–10, Figure 4 · p. 9
5. Conclusion
The conclusion presents text-to-audio grounding as relevant to intelligent vehicle-road cooperation and summarizes CGI’s query graph, cross-modal attention, and cross-gating approach. It reports strong AudioGrounding results and identifies three future directions.
5. Conclusion
- Text-to-audio grounding is framed as relevant to intelligent vehicle-road cooperation because it must model query semantics, emphasize crucial audio snippets, and capture cross-modal interactions.Source: §5 · p. 10
- CGI uses an intra-modal query graph for word relations and attention to create snippet-specific query representations for cross-modal interactions.Source: §5 · p. 10
- CGI adds cross-gating to weaken unimportant audio and query parts, and the paper reports that it outperforms existing methods and designed baselines on AudioGrounding.Source: §5 · p. 10
- Future work will explore Transformer-based cross-modal modeling, graph networks for audio, and lightweight TAG models for edge-cloud collaboration.Source: §5 · p. 10