Source-linked AI summary
LAST: Looped Audio Spectrogram Transformer
Haider Al-Tahan, Sean O'Brien, Anastasia Razdaibiedina, N. Apurva Ratan Murty
TL;DR
Adding transformer layers can improve audio recognition but increases parameters and computation, and recurrence’s contribution to audio recognition is not established. LAST reuses blocks to refine only the class token over fixed audio features. Ten-pass LAST improves on a twelve-layer sequential baseline in AudioSet mAP while using fewer parameters and MACs, with reported robustness and transfer gains.
Problem
Audio recognition requires temporal integration, but adding transformer layers raises parameter and computational costs, and recurrence’s contribution to audio recognition was unknown.
Source: §1 · pp. 1–2Method
LAST processes all tokens on its first pass, then reuses the same blocks to update only the class token over fixed audio features.
Source: §1 · p. 2Results
Ten-pass LAST reaches 0.3450 mAP versus 0.3379 for sequential D=12, a 2.1% relative improvement, with 49.4% fewer parameters and 42% fewer MACs.
Source: §4.1 · p. 3Takeaways & Limitations
LAST’s reported AudioSet gains come with lower computation and parameter counts, and it performs better on the cited frozen-feature classification benchmarks.
Source: §1 · p. 2, §4.1 · p. 3
Abstract
from arXiv · showhide
Increasing depth of transformer models improves recognition, but it comes at a substantial cost. Each additional layer requires more parameters, which makes the process computationally inefficient. We ask whether additional processing can focus on integrating features already computed. Looped Audio Spectrogram Transformer (LAST) first processes all tokens, then reuses the same blocks to refine only the class token over fixed audio features, thereby making later passes inexpensive. On AudioSet, ten-pass LAST achieves 0.345 mean average precision, exceeding a twelve-layer sequential transformer by 2.1% relative with 49.4% fewer parameters, 42% fewer multiply-accumulate operations, and 9.8% higher measured throughput. Across separately trained models, increasing the pass count from two to ten improves accuracy while adding only 1.2% computation. Further evaluations show improved robustness to temporal masking and various other auditory augmentations, with better generalization on classification tasks with music, environmental, and event sounds.
1. INTRODUCTION
The introduction asks whether recurrence can improve audio temporal integration without the parameter cost of added layers. LAST reuses encoder blocks to refine the class token over fixed audio features, and its reported results favor this approach over a deeper sequential model.
1. INTRODUCTION
- Audio recognition requires temporal integration, but adding transformer layers improves recognition at substantial computational and parameter cost.The paper notes that how recurrence contributes to audio recognition was not known.Source: §1 · p. 1
- Unlike CaiT’s separate attention layers, LAST uses existing encoder blocks and adds no parameters.The introduction also distinguishes LAST from approaches that repeatedly update all tokens or attend iteratively to raw inputs.Source: §1 · pp. 1–2, Figure 2 · p. 2
- LAST processes the full spectrogram once, then reuses the same blocks to update only the class token while audio features remain fixed.The paper proposes this after observing that class-token representations are less stable across naive recurrent steps than other spectrogram-token representations.Source: §1 · p. 2
- 2.1% relative mAP improvement over a twelve-layer sequential baseline, with 49.4% fewer parameters and 42% fewer MACs, is reported for ten-pass LAST.The figure compares AudioSet mAP with computation per clip across sequential and recurrent models.Source: Figure 1 · p. 1, §1 · p. 2
- LAST performs better than the sequential baseline on frozen-feature classification benchmarks such as ESC-50 and VGGSound.The supplied result passage names these benchmarks but does not provide metric values.Source: §1 · p. 2
2. METHOD
The method compares sequential processing with shared-block recurrence, then makes recurrent passes inexpensive by fixing audio tokens and updating only the class token.
2.1. Recursion through shared blocks
- A sequential encoder applies D distinct blocks once, whereas a looped encoder reuses D blocks for R passes, keeping parameters fixed by D while full-token recurrence still scales computation with R.Both encoders take spectrogram tokens and a learned class token; a linear head predicts labels from the final class token.Source: §2.1 · p. 2
2.2. Class-token updates in LAST
- LAST updates all tokens on its first pass, then applies attention and the MLP only to the class token while caching audio keys and values for reuse.Training is end to end, with gradients reaching the first pass through the fixed audio features.Source: §2.2 · p. 2
- A cached class-token pass costs O(D(C^2+NC)), versus O(D(NC^2+N^2C)) for a full pass, removing the quadratic-in-N term.The projection term reduces to a single query, so effective depth grows with R at near-constant cost.Source: §2.2 · p. 2
3. EXPERIMENTAL SETUP
The experiments train sequential and recurrent AST variants under a shared AudioSet recipe, then evaluate benchmark accuracy, transfer, throughput, and robustness protocols.
3. EXPERIMENTAL SETUP
- Models are trained from scratch for twelve epochs on AudioSet using 128 Mel bins, 1024 time frames, and sequential depths of 3, 6, 9, or 12 or six shared blocks with 2, 4, 6, 8, or 10 passes.The training set contains 1,908,644 clips; all models use one random seed.Source: §3 · p. 3
3.1. Evaluation
- AudioSet evaluation reports mAP on 18,886 clips, while transfer covers eight classification tasks using logistic regression on frozen class-token features.Inference throughput is benchmarked with bfloat16 and batch size 64.Source: §3.1 · p. 3
- Robustness tests compare a 12-layer sequential model with six-block, ten-pass recurrence under temporal masking and 38 waveform-corruption settings.Masking zeros normalized spectrogram frames selected by attention or uniformly at random; corruptions span twelve families and use the same 4,096 clips.Source: §3.1 · p. 3, Figure 3 · p. 3
4. RESULTS
LAST improves AudioSet accuracy over a deeper sequential transformer with lower parameter and computation costs, and shows stronger results under masking, waveform corruption, and frozen-feature transfer.
4.1. Accuracy and computation
- At R=10, LAST reaches 0.3450 mAP vs 0.3379 for sequential D=12 (2.1% relative improvement), with 49.4% fewer parameters, 42% fewer MACs, and 9.8% higher measured throughput.The deeper sequential model reaches 0.3379 mAP; the comparison reports LAST's resource and throughput advantages at R=10.Source: §4.1 · p. 3
- LAST’s mAP rises at every tested pass count, from 0.3275 at R=2 to 0.3450 at R=10, while eight extra class-token-only passes using cached audio keys and values add just 1.2% computation.Source: §4.1 · p. 3
- With six shared blocks, full-token recurrence falls from 0.3247 mAP at R=2 to 0.3099 at R=10 despite fivefold computation, while requiring approximately 8 times as many MACs as LAST at matched R.The comparison uses separately trained models and does not establish whether an already trained model benefits from additional inference passes.Source: §4.1 · pp. 3–4
4.2. Robustness
- Across 12 waveform-corruption families, LAST has higher mAP in every family; family-averaged mAP is 0.2651 vs 0.2422 for sequential D=12.The comparison covers 38 settings that perturb noise, timing, frequency, reverberation, or amplitude.Source: Figure 4 · p. 4, §4.2 · p. 4
- Mean mAP is 0.1383 vs 0.0905 under attention-based masking and 0.1970 vs 0.1187 under random masking for LAST and sequential AST, respectively.The values integrate accuracy curves over 0–90% masking.Source: §4.2 · p. 4
4.3. Frozen-feature transfer
- LAST outperforms sequential D=12 on five of eight transfer benchmarks, including FMA-Small (43.5% to 50.9%) and DCASE (46.7% to 48.3%).The benchmarks use frozen class-token features and one-vs-rest logistic regression with fixed hyperparameters.Source: Table 1 · p. 4, §4.3 · p. 5