PaperlayerView exact revision on arXiv

Source-linked AI summary

TAC2: Timestamped Audio Captioning as composable functions

Swapnil Bhosale, Prem Seetharaman, Ke Chen, Haonan Chen, Justin Salamon, Zeyu Jin

arXiv:2610.36568v1cs.SD

TL;DR

Real-world audio needs precise, queryable descriptions, but conversational LALMs produce free-form text with limited temporal structure, and prior timestamped captioning is largely fixed and one-shot. TAC2 builds speaker-attributed timelines with weak-teacher fusion and composable IIO tasks, then TAC2-S adds elicited paralinguistic attributes. The resulting representation supports spoken understanding with frozen text-only LLMs and structured supervision for controllable text-to-audio generation.

  • Problem

    Real-world applications need queryable, temporally grounded descriptions of events, speakers, and paralinguistic cues, while conversational LALMs provide free-form text and prior timestamped captioning is largely fixed and one-shot.

    Source: §1 · p. 1

  • Method

    TAC2 fuses complementary weak-teacher predictions into speaker-attributed timelines and models captioning as composable IIO tasks, while TAC2-S elicits paralinguistic attributes with lightweight specialists and distills them into a unified model.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    TAC2-S supports spoken audio understanding with frozen text-only LLMs and provides structured supervision for text-to-audio generation with explicit control over event timing, speaker characteristics, and word-level emphasis.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The structured representation serves as an interface between fine-grained acoustic perception, downstream reasoning, and audio generation.

    Source: §5 · p. 10

    The current schema prioritizes spoken attributes over non-speech and music events, and merges repeated events into spans that can occasionally merge turns.

    Source: A LIMITATIONS AND FUTURE WORK · p. 14

Abstract

from arXiv · show

Real-world audio applications require precise, queryable representations grounded in physical time. Yet modern Large Audio-Language Models (LALMs) increasingly favor conversational generation, producing free-form descriptions that conflate acoustic perception with higher-level reasoning and lack precise temporal grounding. We introduce TAC2, a foundation model that casts timestamped audio captioning as composable functions to construct granular, speaker-attributed acoustic timelines directly from in-the-wild audio. To avoid prohibitively expensive manual annotation, we programmatically fuse complementary predictions from specialized weak teachers through time-registered IoU matching. We model the resulting multi-granularity schema of timestamped events, speaker turns, and fine-grained speech attributes using an Instruction-Input-Output (IIO) completion format, in which each task is a function of the audio, an instruction, and an optional input. A single call decodes the full description autonomously, while composing calls lets earlier outputs or external priors, such as a reference transcript, condition later ones. We further extend TAC2 with expressive speech attributes to obtain TAC2-S, using parameter-efficient specialists to elicit latent paralinguistic capabilities from the pre-trained backbone and distill them into a unified model. We validate the resulting representation in two complementary settings. Paired with frozen text-only LLMs, it yields strong spoken audio understanding across diverse reasoning backends, demonstrating the utility of decoupling fine-grained acoustic perception from downstream reasoning. Used as paired supervision for text-to-audio generation, it supports semantically and temporally grounded reconstruction, together with explicit zero-shot control over event timing, speaker attributes, and word-level emphasis.

1 INTRODUCTION

Real-world audio applications need queryable descriptions grounded in time, but conversational LALMs produce free-form text with limited structural grounding. TAC2 and TAC2-S build speaker-attributed acoustic timelines through weak-supervision fusion and composable IIO tasks, supporting spoken understanding and controllable audio generation.

1 INTRODUCTION

  • Modern LALMs favor conversational generation and free-form descriptions, while applications such as semantic search, precise audio editing, and automated archiving require temporally grounded representations of events, speakers, and paralinguistic cues.The introduction frames this as a mismatch between conversational output and queryable real-world audio representations.Source: §1 · p. 1
  • Existing timestamped captioning grounded descriptions in physical time but largely used fixed, one-shot generation that did not naturally accommodate richer perceptual structure or external priors.The paper asks whether richer descriptions can instead be built by composing simpler captioning functions.Source: §1 · pp. 1–2
  • TAC2 constructs granular, speaker-attributed acoustic timelines from in-the-wild audio by fusing temporally aligned predictions from weak sound-event, diarization, and captioning teachers.The fusion provides structural training targets without manually annotating timelines at scale.Source: §1 · p. 2
  • The IIO format treats each captioning task as a function of audio, an instruction, and optional input, enabling autonomous full-description decoding or calls conditioned on earlier outputs and external priors.A high-quality transcript is one example of an external prior.Source: §1 · p. 2
  • TAC2-S extends TAC2 with expressive speech attributes elicited by lightweight specialists trained on fewer than 10,000 labeled samples, then distilled into a unified model.The specialists annotate the corpus with expressive attributes before distillation.Source: §1 · p. 2
  • TAC2-S paired with frozen text-only LLMs supports spoken understanding, while its structured annotations support text-to-audio training with explicit control over timing, speakers, and word emphasis.The reported settings include diverse reasoning backends and zero-shot generation control.Source: §1 · pp. 2–3

2 RELATED WORK

Related work identifies a gap between high-level audio reasoning and fine-grained acoustic perception, as well as limits in prior timestamped captioning and paralinguistic modeling. TAC2 addresses richer conversational structure with multi-teacher timelines and composable inference, while TAC2-S elicits and consolidates paralinguistic information from a pretrained backbone.

Audio-Language Models and Grounded Perception

  • Prior LALMs show a gap between high-level reasoning and fine-grained perception and remain susceptible to temporal hallucinations and linguistic priors; earlier timestamped captioners primarily ground event boundaries, not speaker-attributed turns or overlapping events.TAC2 instead builds speaker-attributed timelines through multi-teacher temporal fusion and uses IIO for autonomous or externally conditioned inference.Source: §2 · p. 3

Spoken Paralinguistics and Latent Elicitation

  • Prior systems treat expressive speech attributes as isolated categorical tasks or rely on external signal-processing pipelines, whereas TAC2-S uses lightweight specialists to elicit decodable paralinguistic information and distills it into a unified acoustic schema.The approach builds on the possibility that limited parameter adaptation can expose capabilities encoded in pretrained models.Source: §2 · p. 3

3 METHODOLOGY

The methodology builds TAC2’s physical timeline from complementary weak-teacher predictions and trains it through composable IIO tasks. TAC2-S adds elicited paralinguistic attributes using lightweight specialists and distills them into a unified model.

3 METHODOLOGY

  • The methodology has two stages: weak-teacher bootstrapping and IIO training produce TAC2, then specialist elicitation and distillation extend it to TAC2-S.Dense temporal annotation at foundation-model scale is prohibitively expensive, motivating multi-view weak supervision.Source: §3 · p. 3, §3.1 · p. 3

3.1 STAGE 1: PERCEPTUAL FOUNDATION (TAC2)

  • Complementary captioner, diarization, and sound-event predictions are aligned over shared physical time to construct TAC2’s training timelines.Each event takes its temporal span from its temporal owner and attributes from their designated authorities, with a deterministic function resolving multiple candidate values.Source: §3.1 · pp. 3–4, Figure 2 · p. 4
  • IIO decomposes the target schema into instruction, optional teacher-forced input, and output completion tasks, enabling both comprehensive single-call generation and composable multi-pass inference.External priors such as transcripts can condition later attribute-specific tasks.Source: §3.1 · pp. 4–5, Figure 3 · p. 5

3.2 STAGE 2: SPEECH-CENTRIC ELICITATION (TAC2-S)

  • TAC2-S extends TAC2’s timeline with expressive speech attributes, using lightweight specialists to elicit backbone capabilities and distilling their annotations into a unified model.Specialists are trained on small gold-standard datasets and primarily expose latent representations as structured output tags.Source: §3.2 · pp. 5–6
  • The specialist strategy targets latent expressive traits that are strongly linearly decodable in the frozen backbone but less accessible through textual generation.Specialist adaptation leaves early- and mid-layer decodability largely unchanged, with gains concentrated in later layers.Source: §3.2 · pp. 5–6

4 EXPERIMENTS

The experiments evaluate TAC2-S as a structured representation for spoken-audio understanding and as paired supervision for audio generation. Results cover benchmark performance, inference and reasoner robustness, temporal reconstruction, and controllable emphasis.

4.1 AUDIO UNDERSTANDING: VALIDATING SCHEMA FIDELITY

  • 81.9 overall on MMSU, versus 82.9 for TAC2-E; TAC2-S reaches 77.8 on MMSU perception, above Gemini-3.7 Flash (68.7), TAC (63.8), and Qwen3Omni-Captioner (63.5).It also scores 86.4 on MMSU reasoning and remains competitive on MMAR (68.0) and MMAU (75.6).Source: §4.1 · pp. 6–7, Table 1 · p. 7
  • Multi-pass inference lowers MMSU overall from 81.9 to 81.3 versus single-pass, while improving pitch (+12.9), vocal range (+12.0), and accent identification (+9.7).Changing the text reasoner retains similar multi-pass performance: 79.7 with Qwen3-Next and 80.9 with Qwen3.8-Omni.Source: Table 2 · p. 7, §4.1 · p. 7
  • On MSU-Bench, which evaluates speaker-centric understanding in conversational multispeaker English/Mandarin scenarios, TAC2-S’s trace is superior across both comparisons against OmniLLM baselines processing raw audio directly or acting as describers.Source: Table 3 · p. 8
  • Adding individual schema attributes preferentially improves their associated perception tasks while leaving transcript-driven semantic tasks comparatively stable.The figure builds the description attribute by attribute from a transcript-only base.Source: Figure 4 · p. 8

4.2 GENERATION: THE DESCRIPTION AS PAIRED SUPERVISION FOR AUDIO SYNTHESIS

  • The generation evaluation uses TAC2-S captions as paired supervision for a text-to-audio diffusion model, testing whether event timelines reconstruct what happens and when.The illustrated reconstruction includes speech and non-speech events and background sounds such as a censor beep, engine rev, and camera click.Source: §4.2 · p. 8, Figure 5 · p. 9
  • 1.54s PE and 2.14s FLAM temporal error beat timestamp-conditioned PicoAudio2 (2.47s PE; 3.16s FLAM); removing timestamps worsens TAC2-S results to 2.93s PE and 3.16s FLAM.Errors measure mean localizer-peak offset between original and reconstructed audio.Source: Table 4 · p. 9, §4.2 · p. 9
  • Reconstructed audio scores 0.99 gender accuracy, 0.84 age accuracy, 0.69 accent accuracy, 0.77 emotion accuracy, 0.93 speech recall, and 0.14 WER.Gemini-3-Pro rates resemblance to the original at 3.95 and caption adherence at 4.52; Seed Audio 1.0 matches speaker attributes but trails on event timing and both judgments.Source: §4.2 · p. 9
  • Varying a word’s emphasis in the caption makes the synthesized stress peak follow that word and increase with requested intensity, without a dedicated stress-control model.The figure varies emphasis for the same sentence and shows stress probability over time.Source: §4.2 · p. 9, Figure 6 · p. 10

5 CONCLUSION

TAC2 and TAC2-S construct queryable, temporally grounded acoustic representations from in-the-wild audio. The unified representation supports spoken-audio understanding with text-only LLMs and audio reconstruction and control through text-to-audio generation.

5 CONCLUSION

  • TAC2 and TAC2-S construct queryable, temporally grounded acoustic representations from in-the-wild audio.The framework uses multi-view weak supervision, composable IIO functions, and sample-efficient elicitation of paralinguistic attributes.Source: §5 · p. 10
  • Paired with frozen text-only LLMs, the resulting model supports spoken-audio understanding.The conclusion describes the structured timeline as an interpretable interface between perception and reasoning.Source: §5 · p. 10
  • As supervision for text-to-audio generation, the representation supports faithful reconstruction and explicit control over event timing, speaker attributes, and word-level emphasis.Source: §5 · p. 10

AI USE STATEMENT

The authors disclose that generative AI assisted with manuscript language editing and that they reviewed and take responsibility for the final content.

AI USE STATEMENT

  • Generative AI assisted with language editing; the authors reviewed and take responsibility for the final content, including AI-assisted text.The statement attributes all scientific content and conclusions to the authors.Source: AI USE STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

The reproducibility statement describes available model, specialist-corpus, benchmark, and matching resources, while noting that the in-the-wild audio sources are proprietary licensed data. It also directs readers to appendices for implementation and evaluation details.

REPRODUCIBILITY STATEMENT

  • The backbone, specialist gold corpora, and benchmarks are publicly available, but the in-the-wild corpus uses proprietary licensed audio sources.The authors provide complete time-registered IoU matching to reproduce results with alternative audio sources.Source: REPRODUCIBILITY STATEMENT · p. 11

SUMMARY OF APPENDIX

  • The appendices cover implementation, specialist elicitation, IIO tasks, localization, long-form understanding, audio-visual reasoning, generation setup, and prompts.The statement points to Appendices B–I for specified method and evaluation details.Source: REPRODUCIBILITY STATEMENT · p. 11, SUMMARY OF APPENDIX · p. 14

A LIMITATIONS AND FUTURE WORK

TAC2-S prioritizes spoken attributes over non-speech and music, and its temporal schema can merge similar consecutive voices, under-counting speakers. The authors frame these bottlenecks as schema-coverage limits and propose bootstrapping new perceptual axes and teachers, with future work on autonomously evolving descriptions.

A LIMITATIONS AND FUTURE WORK

  • The schema prioritizes spoken attributes over non-speech and music events, limiting performance on benchmarks such as MMAR and MMAU.Repeated events are merged into one span with comma-separated timestamps; similar consecutive voices can also be merged, causing speaker under-counting.Source: A LIMITATIONS AND FUTURE WORK · p. 14
  • The authors characterize these bottlenecks as coverage limits rather than architectural constraints, and propose integrating new perceptual axes and upgraded teachers through teacher-forced bootstrapping.Future work aims to let descriptions evolve autonomously via reinforcement learning, using frozen text reasoners to reward extraction of task-demanded cues.Source: A LIMITATIONS AND FUTURE WORK · p. 14

B IMPLEMENTATION DETAILS

TAC2 is trained on fused timelines from in-the-wild audio, while TAC2-S adds paralinguistic annotations elicited by lightweight specialists and consolidates them into a unified adapter.

B IMPLEMENTATION DETAILS

  • Stage 1 trains TAC2 on approximately two million licensed clips of up to 30 seconds, using timelines fused from sound-event, diarization, and captioning teachers plus forced word alignment.The fusion gates were selected by manual inspection on a small validation set.Source: B IMPLEMENTATION DETAILS · p. 15
  • Stage 2 trains speech specialists on fewer than 10,000 labeled examples per task, using task-specific context and datasets for stress, intonation, emotion, voice profile, pitch, and loudness.A balanced Common Voice accent shard supplements silver voice-profile labels to mitigate their US-accent skew.Source: B IMPLEMENTATION DETAILS · p. 15
  • The specialists are hot-loaded alongside TAC2 over a shared frozen backbone to form TAC2-E, which re-annotates the Stage 1 corpus before a two-pass LLM rewrite reconciles overview and event descriptions.The rewrite step is described as keeping speaker count, gender, and accent consistent.Source: B IMPLEMENTATION DETAILS · p. 15
  • TAC2-S is initialized from Stage 1 TAC2 weights and trained on the enriched corpus with an additional LoRA, mixing the new attribute tasks with the original objectives.After 2,000 steps, the unified adapter consolidates TAC2-E’s modular predictions while retaining TAC2’s structural capabilities.Source: B IMPLEMENTATION DETAILS · p. 15

C ELICITATION: PER-LAYER PROBING

Layer-wise probing finds paralinguistic traits linearly decodable in the frozen backbone, while specialist adaptation mainly improves late-layer readouts and textual generation. It also mitigates systematic output biases, though residual accent errors often involve geographically adjacent regions.

C ELICITATION: PER-LAYER PROBING

  • The probing analysis tests the hypothesis that the backbone already encodes paralinguistic information and specialists improve its accessibility to generation, using independent L2-regularized logistic probes at each of 49 layers.Representations are extracted at the final token for the attribute generation step, and results use five-fold cross-validated AUC.Source: C ELICITATION: PER-LAYER PROBING · p. 15
  • The frozen backbone has peak linear-probe AUCs of 0.89–0.99 across evaluated traits, while specialist adaptation leaves early and middle layers largely unchanged and concentrates its impact in late layers.The table reports mean AUC over depths 0–24 and 36–48, comparing backbone and specialist, plus the specialist’s late-stage lift.Source: C ELICITATION: PER-LAYER PROBING · p. 16, Table 5 · p. 16
  • For relational traits, the specialist yields a +0.10 to +0.12 AUC lift; for accent, already near ceiling internally, the lift is +0.01.The authors interpret the accent result as calibration of the final output distribution rather than improved latent separability.Source: C ELICITATION: PER-LAYER PROBING · p. 16, Table 5 · p. 16
  • The specialist mitigates generation bias and reaches 0.91 regional accent accuracy versus 0.61 for the frozen backbone, although strict 13-class exact match underestimates fidelity when errors are between adjacent regions.The authors relate the mitigation to unchanged early-layer AUCs, distinguishing output generation from the underlying perceptual manifold.Source: C ELICITATION: PER-LAYER PROBING · p. 16, Table 6 · p. 16

D IIO TASKS

TAC2’s IIO interface supports both a comprehensive single-pass description and composed, teacher-forced specialist calls. On long-form audio, TAC2 timelines paired with a frozen text reasoner outperform audio models across evaluated tiers and modes; on conversational speech, TAC2 also localizes pauses competitively with dedicated aligners.

D IIO TASKS

  • IIO training combines isolated attribute tasks with a comprehensive empty-input task, allowing TAC2-S to produce the full schema in one pass or through five composed decodes.In the multi-pass regime, captioned speaker turns and transcript condition four specialist decodes; both regimes yield the same structured schema.Source: D IIO TASKS · pp. 16–17, Table 7 · p. 17

E SEGMENT- AND PAUSE-LEVEL LOCALIZATION

  • The localization test recovers timestamps for transcript segments marked at pause boundaries, using Buckeye windows of at most 20 seconds with 4,947 segments.Segments are bounded by at least 0.1 seconds of silence or an interviewer turn; the evaluation uses eight development speakers and 50 recordings.Source: E SEGMENT- AND PAUSE-LEVEL LOCALIZATION · p. 17
  • For conversational speech, TAC2 reaches 49 ms median onset error, 0.93 boundary agreement at 200 ms tolerance, and the correct segment count in 84% of windows, while remaining competitive with dedicated aligners.Unlike the aligners, TAC2 proposes the segmentation itself.Source: E SEGMENT- AND PAUSE-LEVEL LOCALIZATION · p. 17, Table 8 · p. 18

F LONG-FORM AUDIO UNDERSTANDING

  • TAC2 timelines paired with frozen GPT-5.4 outperform every audio model across all three LongAudioSpan modes and duration tiers; Accuracy nearly matches the dense-caption reference at 79.0 vs. 80.2.For Rubric, correctly placed timestamps for closed-source audio models fall from 42% on the shortest tier to 23% on the longest, while the TAC2 timeline yields 78% and 70%.Source: F LONG-FORM AUDIO UNDERSTANDING · p. 18, Table 9 · p. 19
  • On Chain, answers are 17 points more accurate when the edit is located than when it is missed, a gap larger than for any closed-source audio model, which reaches at most 13 points.The authors attribute the advantage to grounding rather than guessing.Source: F LONG-FORM AUDIO UNDERSTANDING · pp. 18–19

G EXTENDING THE DESCRIBER TO AUDIO-VISUAL REASONING

TAC2-S-V improves the describe-then-reason audio-visual cascade over TAC-V on all three reasoning benchmarks, while its text-to-audio evaluation tests temporal and perceptual reconstruction and the fidelity of recovered attributes.

G EXTENDING THE DESCRIBER TO AUDIO-VISUAL REASONING

  • +1.9, +0.4, and +5.9 over TAC-V on Daily-Omni, WorldSense, and Video-Holmes; TAC2-S-V also beats every native model on Daily-Omni and Video-Holmes.On AVHBench, it leads or matches TAC-V on AVH, VAH, and AVM but trails on AVC; the authors caution that single-reference METEOR tracks phrasing agreement, with even a faithful paraphrase reaching only ~35.Source: G EXTENDING THE DESCRIBER TO… · pp. 19–20, Table 10 · p. 20

H TEXT TO AUDIO MODEL AND TEMPORAL GROUNDING SETUP

  • The text-to-audio generator first learns from structural TAC2 captions, then fine-tunes on enriched TAC2-S captions for speaker-attribute control while retaining temporal fidelity.Temporal evaluation uses 300 held-out English clips with 1,471 events; text-query localizers compare event peak offsets against the original recording, and a text-free frame-cosine measure ranks systems identically to peak-offset metrics and tracks the Gemini-3-Pro judge.Source: H TEXT TO AUDIO MODEL AND TEMPORAL… · pp. 20–21
  • The generator is benchmarked against MiDashengLM-Gen, timestamped and untimestamped PicoAudio2, and Seed Audio 1.0, which is prompted with the same caption format.Source: H TEXT TO AUDIO MODEL AND TEMPORAL… · p. 21

I PROMPTS

  • The prompt section reuses describe-then-reason prompts for several tasks and provides task-specific prompts for MSU-Bench and the text-to-audio judge.Source: I PROMPTS · p. 21

I.1 MSU-BENCH REASONER

  • For MSU-Bench, the reasoner receives TAC2-S’s description in place of audio and otherwise uses the official multiple-choice harness prompt verbatim.Figure 8 shows the describe-then-reason wrapper.Source: §I.1 · p. 21, Figure 8 · p. 21

I.2 TEXT-TO-AUDIO PERCEPTUAL JUDGE (GEMINI-3-PRO)

  • Gemini-3-Pro rates generated audio against the original recording for resemblance and against the shared TAC2-S caption for description fidelity.Figures 9 and 10 show the audio-native judge prompts for the two rubrics.Source: §I.2 · pp. 21–22, Figure 9 · p. 22, Figure 10 · p. 22

Related papers

Loading 2610.36568v1…