Source-linked AI summary
A Transformer-based Audio Captioning Model with Keyword Estimation
Yuma Koizumi, Ryo Masumura, Kyosuke Nishida, Masahiro Yasuda, Shoichiro Saito
TL;DR
AAC has many possible wordings for the same acoustic event or scene, complicating caption estimation and training, while real-world users may not be able to supply keywords. TRACKE estimates keywords from audio and uses them in caption generation; it achieved the highest score among evaluated models without given keywords, with BLEU-1 52.1 and ROUGE-L 34.2.
Problem
One acoustic event or scene can be described with several words, creating many possible captions, while real-world users may not be able to provide keywords in advance.
Source: §1 · p. 1Method
TRACKE combines caption generation with keyword estimation, using an encoder branch to estimate keywords from audio and a decoder that generates captions referring to them.
Source: §1 · p. 1Results
BLEU-1 was 52.1 and ROUGE-L was 34.2 for TRACKE, which achieved the highest score without given keywords among evaluated models.
Source: §4.2 · p. 4Takeaways & Limitations
TRACKE’s results indicate effectiveness for AAC while estimating both captions and keywords.
Source: §5 · p. 4When K is too large, estimated keywords may not reduce word-selection indeterminacy effectively; the authors say K should also be estimated from the input.
Source: §4.2 · p. 4
Abstract
from arXiv · showhide
One of the problems with automated audio captioning (AAC) is the indeterminacy in word selection corresponding to the audio event/scene. Since one acoustic event/scene can be described with several words, it results in a combinatorial explosion of possible captions and difficulty in training. To solve this problem, we propose a Transformer-based audio-captioning model with keyword estimation called TRACKE. It simultaneously solves the word-selection indeterminacy problem with the main task of AAC while executing the sub-task of acoustic event detection/acoustic scene classification (i.e., keyword estimation). TRACKE estimates keywords, which comprise a word set corresponding to audio events/scenes in the input audio, and generates the caption while referring to the estimated keywords to reduce word-selection indeterminacy. Experimental results on a public AAC dataset indicate that TRACKE achieved state-of-the-art performance and successfully estimated both the caption and its keywords.
1. Introduction
AAC can express one sound scene in many ways, making captions difficult to estimate and train. TRACKE combines caption generation with keyword estimation, generating captions conditioned on keywords estimated from audio.
1. Introduction
- AAC can move beyond tagging sounds, scenes, and conditions toward natural-language descriptions with contextual information.The paper illustrates this potential with an alarm described by its location and number of occurrences.Source: §1 · p. 1
- One acoustic event or scene can have many verbal descriptions, producing a combinatorial explosion of possible captions and making ground truth difficult to estimate and training difficult.Source: §1 · p. 1
- Conventional AAC setups use acoustic-event or scene keywords to restrict caption choices, but smart-speaker users may not know which keywords to provide.Source: §1 · p. 1
- Because real-world users may not be able to provide keywords in advance, AAC needs to estimate them while generating captions.The paper frames this as jointly performing AAC and acoustic event detection or acoustic scene classification.Source: §1 · p. 1
- TRACKE estimates keywords with an encoder branch and generates captions while referring to those estimates to reduce word-selection indeterminacy.During training, ground-truth keywords are extracted from the ground-truth caption and used to train the keyword-estimation branch.Source: §1 · p. 1
- TRACKE decomposes AAC into caption generation and keyword estimation, adopting weakly supervised polyphonic acoustic event detection for the latter.The authors describe this as the first study to apply a Transformer to AAC.Source: §1 · p. 1
2. Preliminaries of audio captioning
AAC maps acoustic features to a word sequence, commonly through an encoder-decoder process. Keywords can serve as an attention factor to select among words describing an acoustic event or scene.
2. Preliminaries of audio captioning
- AAC translates an input audio-feature sequence into an output word sequence.The output consists of word indices from a vocabulary.Source: §2 · pp. 1–2
- A conventional method uses scene-aware acoustic features, whereas this work explicitly addresses word-selection indeterminacy through acoustic event detection or scene classification.Source: §2 · p. 1
- Transformer use had been investigated for acoustic event detection and scene classification, while earlier AAC studies used sequence-to-sequence models.Source: §2 · pp. 1–2
- A sequence-to-sequence AAC model encodes the input as a feature representation, then predicts each word from that representation and preceding output words.The decoder estimates each next-word posterior recursively and uses beam search to select words.Source: §2 · p. 2
- Because one acoustic event or scene can have several descriptions, AAC faces word-selection indeterminacy and a combinatorial explosion of possible captions.Previous studies used keywords as metadata to reduce this indeterminacy.Source: §2 · p. 2
- Passing keywords to the decoder is expected to make them an attention factor for selecting among possible words corresponding to an acoustic event or scene.The keyword set m contains K words.Source: §2 · p. 2
3. Proposed Model
TRACKE estimates keywords from audio and uses them when generating captions, addressing word-selection indeterminacy without requiring users to supply keywords. Its model combines captioning and keyword-estimation objectives, with keyword labels extracted from training captions.
3. Proposed Model
- TRACKE estimates keywords from audio rather than requiring users to supply them, broadening the proposed AAC use case.The authors note that users who can provide keywords may already know the sound environment.Source: §3 · p. 2
3.1. Model overview
- TRACKE concatenates embedded estimated keywords with encoder features, then uses the combined representation as decoder-attention keys and values.The model uses VGGish bottleneck features for audio embedding and fastText for caption-word and keyword embeddings.Source: §3.1 · p. 2
3.2. Keyword-estimation branch
- Because event durations vary and labels lack timestamps, M estimates framewise keyword posteriors, max-pools over time, and selects top-K keywords.Sorting and top-K selection disconnect M from decoder loss, so keyword-branch parameters are updated only by keyword-estimation accuracy.Source: §3.2 · pp. 2–3
3.3. Rule-based keyword extraction for training
- Training keywords are frequent lemmas of nouns, verbs, adjectives, and adverbs that appear in the vocabulary built from training captions.The vocabulary contains the most frequent C lemmas, excluding “be”.Source: §3.3 · p. 3
3.4. Training procedure
- Training jointly minimizes caption cross-entropy and prior-weighted binary cross-entropy for keyword estimation.The keyword-loss weights are reciprocals of the keyword and non-keyword prior probabilities.Source: §3.4 · p. 3
4. Experiments
Experiments evaluate TRACKE on Clotho against AAC baselines and keyword-oracle variants; it achieves the highest score without supplied keywords.
4.2. Results
- TRACKE achieved the highest score without given keywords, reaching 52.1 BLEU-1 and 34.2 ROUGE-L versus Oracle1’s 53.4 BLEU-1 and 35.1 ROUGE-L.Those scores were 97.6% and 97.4% of Oracle1’s respective scores; Table 1 reports the Clotho results.Source: §4.2 · p. 4, Table 1 · p. 4
- Oracle2 significantly outperformed TRACKE, whose keyword-estimation accuracy was 48.1%.The reported percentage is the share of estimated keywords included in the ground-truth keywords.Source: §4.2 · p. 4
- When K is too large, estimated keywords may not reduce word-selection indeterminacy; Transformer and TRACKE with K = 10 scored almost the same.The authors identify estimating K from the input as a further improvement direction.Source: §4.2 · p. 4
- Transformer was slightly better than LSTM, but its effectiveness for AAC remains to be affirmed using more large-scale datasets.The authors note that Transformer training requires a large-scale dataset.Source: §4.2 · p. 4
- The authors attribute the large performance gaps between the baseline and other models to pre-trained models, naming VGGish and fastText.They identify pre-trained models as the major difference among the compared approaches.Source: §4.2 · p. 4
- Figure 3 examples show estimated keywords guiding captions, while keyword posteriors also reflect implicit co-occurrence, including a rise in “person” without human sounds.The authors suggest this may reflect that people usually open and close doors.Source: §4.2 · p. 4, Figure 3 · p. 4
5. Conclusions
TRACKE combines audio caption generation with keyword estimation: it estimates keywords from the input audio and generates captions using them. Experiments indicate its effectiveness for AAC, while future work targets improved keyword estimation and keyword-guided generation strategies.
5. Conclusions
- TRACKE estimates keywords from audio and generates captions while referring to them to address AAC word-selection indeterminacy.The authors report that experimental results indicate the model's effectiveness for AAC.Source: §5 · p. 4
- The keyword-estimation branch uses weakly supervised polyphonic AED, with ground-truth keywords extracted from captions by a heuristic rule.Source: §5 · p. 4
- Future work includes improving keyword estimation while adopting keyword-guided generation strategies from NLP and image captioning.Source: §5 · p. 4