PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Textual Chain-of-Thought: JEPA-Conditioned Latent Reasoning for Large Audio Language Models

Donghang Wu, Haoyang Zhang, Yizhou Peng, Shreyas Gopal, Yi-Wen Chao, Chen Chen, Hexin Liu, William Tjhi, Eng-Siong Chng

arXiv:2609.34407v1eess.AS

TL;DR

Text-based audio CoTs may not be explicitly aligned with acoustic evidence. JELAR instead trains LALMs to predict latent targets conditioned on WavJEPA representations, and reports average-accuracy gains of 2.70 and 9.10 percentage points over Audio-Reasoner on MMAU-mini and MMAR, respectively.

  • Problem

    Audio LALM reasoning often relies on textual CoTs synthesized from captions, whose alignment with acoustic evidence is not ensured despite task-relevant audio details.

    Source: §1 · p. 1

  • Method

    JELAR replaces explicit textual CoT supervision with continuous latent reasoning targets built by answer-aware queries attending to frozen WavJEPA waveform representations.

    Source: §1 · p. 2

  • Results

    2.70 and 9.10 absolute percentage-point average-accuracy gains over Audio-Reasoner on MMAU-mini and MMAR, respectively, were reported for JELAR.

    Source: §1 · p. 2, §4 · p. 4

  • Takeaways & Limitations

    Controlled comparisons and ablations support JEPA-conditioned latent reasoning as an alternative to explicit textual CoT supervision in these LALM evaluations.

    Source: §1 · p. 2, §4 · p. 4

Abstract

from arXiv · show

Explicit textual Chain-of-Thought (CoT) has improved the reasoning ability of large audio language models (LALMs). However, textual CoTs are often constructed from text captions of audio and provide limited access to the acoustic evidence, which can introduce problems like hallucination. To address this modality-gap issue, we introduce JELAR, a Joint-Embedding Predictive Architecture (JEPA)-based latent reasoning framework that conditions latent reasoning supervision on acoustic representations learned from raw waveforms. During training, a frozen WavJEPA model provides representations learned directly from raw waveforms. A non-causal expert first constructs answer-aware queries, which cross-attend to WavJEPA embeddings to produce latent reasoning targets. The LALM is trained to predict these targets before generating its response. Experimental results show that JELAR improves the Audio-Reasoner baseline by 2.70 and 9.10 absolute percentage points on MMAU-mini and MMAR, respectively, demonstrating the effectiveness of JEPA-conditioned latent reasoning as an alternative to explicit textual CoT supervision.

1. INTRODUCTION

Audio reasoning must integrate acoustic evidence across time, but textual CoT can be weakly aligned with the audio. JELAR uses waveform-derived JEPA representations to condition latent reasoning targets.

1. INTRODUCTION

  • Textual CoTs, such as Audio-Reasoner’s, are often synthesized from captions, so their alignment with audio evidence is not ensured and may contribute to hallucinations.Task-critical prosody, timbre, overlapping events, temporal structure, background acoustics, and cross-source relationships may not be faithfully reduced to text.Source: §1 · p. 1
  • Latent reasoning alone may still encode linguistic or shortcut information without acoustic targets, and many LALM audio encoders, including Whisper, are optimized for audio-to-text prediction.The stated challenge is to supervise latent reasoning with representations derived directly from acoustics rather than textual reasoning traces.Source: §1 · p. 1
  • JEPAs predict latent targets from contextual observations rather than reconstructing inputs or predicting text, while WavJEPA [14] learns temporally predictable waveform structure without transcript, caption, or answer supervision.These representations motivate an acoustic target space for latent reasoning trajectories.Source: §1 · pp. 1–2
  • JELAR improves average accuracy over the Audio-Reasoner [8] baseline by 2.70 absolute percentage points on MMAU-mini and 9.10 on MMAR.Controlled experiments also report consistent gains over explicit CoT training.Source: §1 · p. 2
  • A frozen WavJEPA [14] encoder and training-only non-causal expert produce fixed-length targets from answer-aware queries, which the LALM predicts before answering.At inference, the LALM generates the latent sequence and textual answer without the expert or WavJEPA.Source: §1 · p. 2
  • JELAR replaces explicit textual CoT supervision with continuous latent targets conditioned on JEPA-based waveform representations.The framework uses WavJEPA [14] representations learned directly from raw waveforms.Source: §1 · pp. 1–2

2. METHOD

JELAR trains an LALM to predict continuous latent reasoning embeddings built from answer-aware queries over raw-waveform representations. At inference, the LALM recursively feeds back its own latent predictions before generating a textual response.

2.1. Overview

  • The LALM predicts a fixed-length sequence of continuous latent embeddings, using expert-provided inputs during training and recursively feeding back its own predictions at inference.Each predicted embedding becomes the input for the next reasoning step.Source: §2.1 · p. 2

2.2. Supervised Fine-Tuning (SFT) for Latent Reasoning

  • A bidirectional BERT expert forms answer-aware queries from the question and ground-truth answer, which attend to frozen WavJEPA waveform embeddings [14] and are projected into the LALM embedding space.The expert constructs the latent reasoning targets because continuous reasoning trajectories are unavailable in the training data.Source: §2.2 · p. 2
  • The latent targets are inserted between special tokens during teacher-forced training, and response loss also optimizes the non-causal expert to construct targets that support answer generation.WavJEPA remains frozen while the expert, latent prediction head, audio encoder and projector, and LLM backbone are jointly optimized.Source: §2.2 · pp. 2–3

2.3. Inference

  • At inference, JELAR discards the non-causal expert and WavJEPA, generates K latent embeddings recursively from the audio and question, then decodes the textual response autoregressively.After the K latent steps, the model appends the closing latent token before standard response generation.Source: §2.3 · p. 3

3. EXPERIMENTS

JELAR improves on Audio-Reasoner in average accuracy on both benchmarks, with a larger gain on MMAR. Ablations show benefits from acoustic conditioning and WavJEPA targets, while adding textual CoT after latent reasoning reduces performance.

3.1. Experimental Setup

  • JELAR uses the Audio-Reasoner backbone and training configuration but replaces explicit textual CoT supervision with continuous latent reasoning.Evaluation uses MMAU-mini, with closed-choice sound, music, and speech questions, and MMAR, with human-curated multiple-choice multi-step audio reasoning questions; the latent stage uses K = 20 embeddings, selected on held-out development data.Source: §3.1 · p. 3

3.2. Main Results

  • JELAR raises MMAU-mini average accuracy from 59.70% to 62.40% and MMAR average accuracy from 43.70% to 52.80%, a 9.10-percentage-point MMAR gain.On MMAU-mini, speech accuracy rises from 51.37% to 60.66%, while music improves slightly and sound accuracy decreases; on MMAR, JELAR outperforms Audio-Reasoner across all seven categories.Source: §3.2 · pp. 3–4, Table 1 · p. 4, Table 2 · p. 4

3.3. Ablation study

  • Increasing K from 8 to 20 raises MMAU-mini accuracy from 60.60% to 62.40% and MMAR accuracy from 49.60% to 52.80%, with both peaking at K = 20.Increasing K further yields no additional gains, indicating that more latent reasoning steps are not inherently beneficial once sufficient reasoning capacity is reached.Source: Table 3 · p. 4, §3.3 · p. 4
  • Removing acoustic conditioning yields 60.50%/47.60%; native Whisper conditioning yields 61.60%/50.70%; WavJEPA yields 62.40%/52.80% on MMAU-mini/MMAR, respectively.Acoustic conditioning adds benefit beyond answer-aware latent supervision, and WavJEPA is more effective than the native Whisper encoder in this setting.Source: Table 4 · p. 4, §3.3 · p. 4
  • Adding explicit textual CoT after latent reasoning lowers accuracy from 62.40% to 61.40% on MMAU-mini and from 52.80% to 49.30% on MMAR.The ablation supports latent-only reasoning as the more effective format in this setting.Source: §3.3 · p. 4

4. CONCLUSION

JELAR replaces explicit textual CoT supervision with latent targets conditioned on frozen WavJEPA representations. Under controlled comparisons with Audio-Reasoner, it improves average accuracy on MMAU-mini and MMAR.

  • JELAR improves average accuracy by 2.70 percentage points on MMAU-mini and 9.10 percentage points on MMAR versus Audio-Reasoner in controlled comparisons.Source: §4 · p. 4
  • JELAR replaces explicit textual CoT supervision with continuous latent targets constructed by answer-aware semantic queries attending to frozen WavJEPA representations.Source: §4 · p. 4
  • Ablations support acoustic conditioning, WavJEPA-based target representations, and latent-only reasoning; future work will investigate larger backbones and more diverse audio reasoning data.Source: §4 · p. 4
Loading 2609.34407v1…