Source-linked AI summary
Audio Captioning Transformer
Xinhao Mei, Xubo Liu, Qiushi Huang, Mark D. Plumbley, Wenwu Wang
TL;DR
Audio captioning must identify audio events and express their spatial-temporal relationships, while recurrent models can be limited in long-term temporal modeling. This paper proposes ACT, a convolution-free Transformer evaluated on AudioCaps, where it shows competitive performance and is reported to outperform state-of-the-art systems in most metrics.
Problem
Audio captioning must identify audio events and express their spatial-temporal relationships, while RNNs can be limited in modeling long-term temporal dependencies.
Source: §1 · p. 1Method
ACT is a convolution-free encoder-decoder Transformer that applies self-attention to time-axis patches of log mel-spectrograms.
Source: §1 · p. 1, §3.1 · p. 2Results
Takeaways & Limitations
The authors conclude ACT models global and fine-grained audio information simultaneously and captures temporal relationships better than CNNs.
Source: §6 · p. 4Further research is needed to adapt ACT for audio clips of varied lengths.
Source: §6 · p. 4
Abstract
from arXiv · showhide
Audio captioning aims to automatically generate a natural language description of an audio clip. Most captioning models follow an encoder-decoder architecture, where the decoder predicts words based on the audio features extracted by the encoder. Convolutional neural networks (CNNs) and recurrent neural networks (RNNs) are often used as the audio encoder. However, CNNs can be limited in modelling temporal relationships among the time frames in an audio signal, while RNNs can be limited in modelling the long-range dependencies among the time frames. In this paper, we propose an Audio Captioning Transformer (ACT), which is a full Transformer network based on an encoder-decoder architecture and is totally convolution-free. The proposed method has a better ability to model the global information within an audio signal as well as capture temporal relationships between audio events. We evaluate our model on AudioCaps, which is the largest audio captioning dataset publicly available. Our model shows competitive performance compared to other state-of-the-art approaches.
1. INTRODUCTION
Audio captioning must identify audio events and express their spatial-temporal relationships, but existing CNN and RNN approaches have limitations in modeling temporal information. The paper proposes a convolution-free, self-attention-based Transformer and reports competitive AudioCaps performance.
1. INTRODUCTION
- Audio captioning must identify audio events and express their spatial-temporal relationships, supporting uses from hearing assistance to multimedia retrieval and security surveillance.Source: §1 · p. 1
- RNNs can be limited in long-term temporal modeling, while CNN convolution is local and can require deep, hard-to-train layers that risk over-fitting.Source: §1 · p. 1
- Earlier captioning systems replaced RNN decoders with Transformers, and CNN+Transformer models had achieved state-of-the-art performance in the area.Source: §1 · p. 1
- ACT is a convolution-free Transformer that splits log mel-spectrograms into time-axis patches, allowing each patch to attend to all others and model global long-range dependencies.The authors also describe its features as fine-grained because the model does not down-sample.Source: §1 · p. 1
- Because Transformers usually require more training data than CNNs, ACT pre-trains its encoder on AudioSet as an audio-tagging task to improve generalization.A class token represents global clip information, so the decoder can attend to both local and global audio information.Source: §1 · p. 1
- ACT shows competitive performance against other state-of-the-art methods on AudioCaps.Source: §1 · p. 1
2. RELATED WORK
Related AAC work developed encoder-decoder systems and explored ways to control or improve caption generation. The authors position ACT within the broader adoption of Transformers in language and vision tasks.
2. RELATED WORK
- Early AAC work used an RNN encoder-decoder with an alignment model, while specificity was introduced to guide caption content.Source: §2 · p. 1
- AAC approaches also explored multi-scale encoders and semantic attention, CNN audio encoders, Transformer decoders [13] [14], multitask learning [15], retrieval [16], and reinforcement learning.Source: §2 · p. 2
- Following attention-only image Transformers and image captioning work [10] [20], the authors propose ACT as a Transformer for audio captioning, which they say had not previously been done.Source: §2 · p. 2
3. PROPOSED METHOD
ACT is a convolution-free sequence-to-sequence Transformer that maps an audio clip’s log-mel spectrogram to word probabilities. Its encoder represents audio patches with self-attention, while its decoder generates captions using masked self-attention and attention to encoder features.
3. PROPOSED METHOD
- ACT takes a log-mel spectrogram as input and outputs posterior probabilities for predicted words in a convolution-free sequence-to-sequence model.Figure 1 places the encoder on the left and decoder on the right.Source: §3 · p. 2, Figure 1 · p. 2
3.1. Encoder
- The encoder splits the log-mel spectrogram into non-overlapping time patches, projects them into embeddings, and prepends a learnable class token for clip-global information.Trainable positional embeddings are added to the patch embeddings.Source: §3.1 · p. 2
- The encoder stacks multi-head self-attention and feed-forward sublayers, using layer normalization and residual connections around each sublayer.Its architecture follows ViT and DeiT with 12 encoder blocks, 12 heads, and a 768-dimensional embedding.Source: §3.1 · p. 2
3.2. Decoder
- The decoder embeds caption words, applies masked self-attention and cross-attention to encoder audio features, then predicts a vocabulary distribution.Word vectors are pretrained with Word2Vec on the caption corpus.Source: §3.2 · pp. 2–3
- Training minimizes cross-entropy with teacher forcing, and the evaluated ACT variants share an encoder but differ in decoder layer and head counts.Table 1 summarizes the decoder variants’ parameters.Source: §3.2 · p. 3, Table 1 · p. 3
4. EXPERIMENTS
The experiments use AudioSet for encoder pre-training and AudioCaps for caption evaluation, with log-mel patches as input. They also describe the training setup and the metrics used to assess captions.
4.1.1. AudioSet
- AudioSet contains over 2 million 10-second YouTube clips labeled across 527 sound classes; 1,934,187 training and 18,887 evaluation clips were available.Each clip can have one or more sound-event labels.Source: §4.1.1 · p. 3
4.1.2. AudioCaps
- AudioCaps contains around 50k AudioSet-sourced clips, with one human caption per training clip and five per validation and test clip.Source: §4.1.2 · p. 3
4.2. Data pre-processing
- Audio clips are converted to 32 kHz and padded to 10 seconds, then represented as log-mel spectrograms split into 125 non-overlapping 64 × 4 patches.SpecAugment is applied during training; captions are lowercased, stripped of punctuation, and marked with start and end tokens.Source: §4.2 · p. 3
4.3. Audio tagging pre-training
- The encoder is initialized from DeiT weights and pre-trained on AudioSet sound tagging to address limited audio-captioning training data.The pre-trained encoder reaches 0.43 mAP on AudioSet’s evaluation set.Source: §4.3 · p. 3
4.4. Experimental setups
- The captioning model is trained for 30 epochs with Adam, dropout, label smoothing, and beam search at inference.Training uses batch size 32; the learning rate warms up to 1 × 10^-4 and is reduced by a factor of 0.1 every 10 epochs.Source: §4.4 · p. 3
4.5. Evaluation metrics
- Evaluation combines machine-translation metrics BLEU, ROUGE-L, and METEOR with captioning metrics CIDEr, SPICE, and SPIDEr.SPIDEr averages SPICE and CIDEr and is the DCASE challenge’s official ranking metric.Source: §4.5 · pp. 3–4
5. RESULTS
On AudioCaps, ACT is competitive with prior systems, with stronger machine-translation metrics than CNN+Transformer but a slightly lower CIDEr score. Ablations show the value of encoder pre-training and modest sensitivity to decoder settings.
5.1. Performance comparison
- ACT outperforms RNN+RNN on all metrics, exceeds CNN+RNN on most, and beats CNN+Transformer on machine-translation metrics while scoring slightly lower on CIDEr.ACT takes under five minutes per epoch in the reported experiments, versus seven minutes for CNN+Transformer.Source: §5.1 · p. 4
5.2. Ablation studies
- DeiT initialization improves ACT across all metrics, AudioSet tagging pre-training further approaches state-of-the-art performance, and ACT trained without encoder pre-training trails CNN+Transformer.Source: §5.2 · p. 4
- Decoder variants perform similarly and ACT is slightly sensitive to layer and head choices; four-layer ACT m scores better on machine-translation metrics, while ACT l scores higher on CIDEr and SPIDEr.The paper attributes the use of shallow decoders partly to audio-captioning data being smaller and captions shorter than in natural-language tasks.Source: §5.2 · p. 4
6. CONCLUSION
The conclusion presents ACT as a self-attention-only audio captioning model that jointly represents global and fine-grained signal information. It reports stronger temporal-relation modelling than CNNs and better scores than other systems on most metrics, while identifying varied clip lengths as future work.
6. CONCLUSION
- ACT uses a full Transformer with self-attention to model global and fine-grained audio information and capture temporal relations between events better than CNNs.The authors report that ACT outperforms other state-of-the-art audio captioning systems in most metrics and call for future adaptation to clips of varied lengths.Source: §6 · p. 4