Source-linked AI summary
Thinking While Listening: Simple Test Time Scaling For Audio Classification
Prateek Verma, Mert Pilanci
TL;DR
The paper asks whether reasoning over audio-category predictions can bring test-time scaling to audio classification. It builds patch-level reasoning traces for existing and newly designed models, and reports consistent gains as trace sampling increases, including from a frozen GPT-2 adapted by retraining its embedding matrix.
Problem
The paper asks how to add category-space reasoning to audio classification and whether an architecture can support thinking and test-time scaling.
Source: §1 · p. 1Method
The approach samples patch-level category predictions into reasoning traces and aggregates them with frozen reasoning models, including a GPT-2 model with a retrained embedding matrix.
Source: §2.3 · p. 3Results
Classification performance improves consistently as posterior sampling produces longer traces, and the embedding-tuned frozen GPT-2 setup improves over open-source reasoning models using complex prompts.
Source: §2.3 · p. 3, §4 · p. 4, §5 · p. 4Takeaways & Limitations
The results show test-time performance scaling across two audio benchmarks while keeping the backbone weights and input audio fixed.
Source: §5 · p. 5
Abstract
from arXiv · showhide
We propose a framework that enables neural models to "think while listening" to everyday sounds, thereby enhancing audio classification performance. Motivated by recent advances in the reasoning capabilities of large language models, we address two central questions: (i) how can thinking be incorporated into existing audio classification pipelines to enable reasoning in the category space and improve performance, and (ii) can a new architecture be designed from the ground up to support both thinking and test-time scaling? We demonstrate that in both settings, our models exhibit improved classification accuracy. Leveraging test-time scaling, we observe consistent gains as the number of sampled traces increases. Furthermore, we evaluate two open-source reasoning models, GPT-OSS-20B and Qwen3-14B, showing that while such models are capable of zero-shot reasoning, a lightweight approach--retraining only the embedding matrix of a frozen, smaller model like GPT-2--can surpass the performance of billion-parameter text-based reasoning models.
1. INTRODUCTION AND RELATED WORK
The paper adapts test-time scaling to audio classification by reasoning over sequences of patch-level category predictions. This offers an alternative to scaling model or dataset size and to transforming inputs, with classification performance improving while the model and audio remain fixed.
1. INTRODUCTION AND RELATED WORK
- Unlike approaches that verify sampled outputs and select the best one, this method uses an LLM to consume patch-level prediction sequences and output categories directly.The approach draws on chain-of-thought prompting and self-consistency, but constructs its reasoning steps from perceptual-model outputs rather than textual exemplars.Source: §1 · p. 1
- Test-time scaling can improve audio classification with a fixed model and input by aggregating stochastic patch predictions through a reasoning model.The patch-level category trace reflects the classifier’s evolving hypothesis over time, with patches serving as evidence for the final category.Source: §1 · p. 1
- Audio classification has traditionally gained performance by scaling model parameters or dataset size, while test-time strategies such as multi-level augmentation have also improved audio-captioning metrics.The paper positions reasoning over audio patch predictions as a different test-time scaling strategy.Source: §1 · p. 1
2. METHODOLOGY
The methodology turns audio classification into a process of generating patch-level category traces and reasoning over them at inference. It adapts existing audio models and describes a new setup that samples traces for aggregation by frozen reasoning models.
2.3. Test Time Scaling Over Reasoning Traces
- Patchwise category probabilities are sampled repeatedly from a frozen audio classifier to form a reasoning trace, which a frozen LLM uses to refine the final classification.Inference keeps the audio fixed; trace length is controlled by the number of samples drawn per patch.Source: Figure 1 · p. 2, §2.3 · p. 3
2.1. Reasoning Traces From Existing Classification Model
- For existing AST and YAMNet models, causal predictions across audio patches form category traces; longer context yields more confident and accurate classifications.For 5-second ESC-50 audio, the method uses 500 ms patches; YAMNet is processed incrementally to generate predictions at each step.Source: §2.1 · p. 2
2.2. Models That Can Think While Listening
- Rather than training against a single target vector, the method predicts category probabilities per patch during training with a sigmoid layer and minimizes mean-squared-error loss.Because each 1-second FSD-50K segment may contain multiple categories, each MLP head’s 200-dimensional output is normalized as a probability distribution for patchwise sampling.Source: §2.2 · p. 3
2.3. Test Time Scaling Over Reasoning Traces
- For YAMNet and AST, 500 ms chunk predictions are sampled from posterior distributions and aggregated; sampling length T produces a sequence of 2P × T tokens with confidence for P patches.The sequences can be given to GPT-OSS 20B or Qwen3 14B with a structured prompt for zero-shot reasoning.Source: §2.3 · p. 3
- A frozen GPT-2 model is adapted by retraining only its embedding matrix, and this setup improves performance over open-source reasoning models using complex prompts.The category and confidence tokens form the input trace; GPT-2’s weights and positional embeddings remain fixed.Source: §2.3 · p. 3
3. DATASET
The study evaluates audio classification on ESC-50 and FSD-50K, covering single-label and multi-label settings with dataset-specific metrics.
3. DATASET
- ESC-50 has 50 categories and 2,000 five-second clips with one label each; FSD-50K predicts 200 categories from one-second segments that may contain multiple labels.The study reports top-1 accuracy for ESC-50 and AUC for FSD-50K.Source: §3 · p. 3
4. EXPERIMENTS AND RESULTS
Longer patch-level sampling traces improve audio classification results across the evaluated backbones and datasets. On FSD-50K, eight samples per patch surpass the Audio Transformer baseline; on ESC-50, frozen GPT-2 with a retrained embedding matrix outperforms the evaluated larger reasoning models.
4. EXPERIMENTS AND RESULTS
- GPT-OSS outperforms Qwen on this task, but neither surpasses frozen GPT-2 with a retrained embedding matrix on ESC-50.The reported comparison keeps audio backbones and reasoning-model transformer blocks fixed.Source: §4 · pp. 3–4
- On ESC-50, frozen YAMNet and AST are evaluated with an 84% frozen-model baseline, while full AST fine-tuning reaches 88.8%; test-time scaling can bring a frozen model to full-tuned performance.Fig. 2 shows top-1 accuracy increasing with trace length for frozen YAMNet and AST.Source: §4 · p. 3, Figure 2 · p. 4
- Longer traces improve both ESC-50 backbones, and eight samples per patch surpass the Audio Transformer baseline on FSD-50K.The reported FSD-50K comparison uses 1-second chunks; Fig. 3 shows the trace-length results.Source: §4 · pp. 3–4, Figure 3 · p. 4
- Short FSD-50K traces do not improve over the Audio Transformer baseline, while increasing sampling from 1 to 2 to 8 per patch boosts performance.The eight-sample setting surpasses the baseline on 1-second inputs.Source: §4 · p. 4, Figure 3 · p. 4
- Temperature adjustment gives only a 0.2% gain, which the authors consider insufficient to justify a larger study in this setup.Table 2 covers ESC-50 accuracy across temperatures and trace lengths with frozen AST and GPT-2.Source: §4 · p. 4, Table 2 · p. 4
5. CONCLUSION AND FUTURE WORK
The paper presents test-time scaling for audio classification through reasoning traces, both by extending pretrained models and by designing an architecture for trace sampling. Its results point to patch-level category reasoning as a complementary way to improve performance while keeping model weights and audio input fixed.
5. CONCLUSION AND FUTURE WORK
- The approach extends pretrained audio models and introduces a new architecture that samples reasoning traces conditioned on audio.The paper describes the traces as sequences of category predictions and reports gains as trace length increases.Source: §5 · p. 4
- Retraining only GPT-2’s embedding matrix and allowing multiple category predictions per patch yield consistent performance gains as sampling traces lengthen.The paper also reports ablations integrating text-based reasoning.Source: §5 · p. 4
- Across two audio benchmarks, performance can be scaled by reasoning over patch-level category traces while keeping backbone weights and input audio fixed.The authors describe this as complementary to scaling parameters or training data.Source: §5 · pp. 4–5